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Foreword 


IEA's mission is to enhance knowledge about education systems worldwide and to 
provide high-quality data that will support education reform and lead to better 
teaching and learning in schools. In pursuit of this aim, it conducts, and reports on, 
major studies of student achievement in literacy, mathematics, science, citizenship, 
and digital literacy. IEA studies, most notably TIMSS, PIRLS, ICCS, and ICILS, 
have set the benchmark for international comparative studies in education. 

These well-established studies have generated vast datasets encompassing 
student achievement, disaggregated in a variety of ways, along with a wealth of 
contextual information which contains considerable explanatory power. The numerous 
reports that have emerged from them represent an invaluable contribution to the 
corpus of educational research. 

Nevertheless, IEA's goal of supporting education reform needs something more: 
deep understanding of education systems and the many factors that bear on student 
learning advances through in-depth analysis of the global datasets. IEA has long 
championed such analyses and facilitates scholars and policymakers in conducting 
secondary analysis of the datasets. So IEA provides software such as the International 
Database Analyzer to encourage the analysis of their datasets, support numerous 
publications, including a peer-reviewed journal — Large-Scale Assessment in Education 
— dedicated to the science of large-scale assessment and publishing articles that 
draw on large-scale assessment databases; and organize a biennial international 
research conference to nurture exchanges between researchers working with 
IEA data. 

The IEA Research for Education series represents a further effort by IEA to 
capitalize on these unique datasets, so as to provide powerful information for 
policymakers and researchers. Each report focuses on a specific topic and is 
produced by a dedicated team of leading scholars on the theme in question. Teams 
are selected on the basis of an open call for tenders; there are two such calls a year. 
Tenders are subject to a thorough review process, as are the reports produced. (Full 
details are available on the IEA website.) 


vi Foreword 


This fifth volume in the series is concerned with socioeconomic inequality and 
educational outcomes. Socioeconomic status is a key variable in social science 
research. It is especially important to the understanding of educational inequality 
and how best to address it. There is a substantial literature on the links between 
students' educational achievement and their family background. Despite this, 
challenges in measuring socioeconomic status and identifying its impact persist. 

This book draws on data collected over 20 years in the IEA Trends in 
International Maths and Science Study (TIMSS) and scrutinizes student 
achievement levels in relation to their socioeconomic status. Besides achievement 
data, TIMSS has been collecting background information from students, teachers, 
and school principals. Using a modified version of the TIMSS home educational 
resources index, the authors have identified tentative patterns in the changes over 
time. Specifically, they have established which countries have seen greater 
educational inequality attributable to family background and which have seen a 
reduction. They also identify which countries have managed to increase the 
academic performance of disadvantaged students over the period and those which 
have not. 

There are no easy answers to the challenges posed by the educational 
underachievement of students from disadvantaged backgrounds. It remains, 
however, one of the most significant issues facing societies and their education 
systems. While family background is a critical variable, the authors properly point 
out that macro-level factors such as gross national wealth per person, total 
expenditure on education, and degree of centralization of the education system all 
play a part. What this book does offer is a data-driven focus on the effects of 
socioeconomic status on educational outcomes and a methodology for deeper 
national investigation across the many cycles of TIMSS. Both researchers and 
policymakers will find it suggestive in terms of exploring national contexts more 
precisely and devising policy actions to ameliorate educational underachievement. 

Future publications in the series will examine trends to evaluate the success of 
countries in strengthening teacher quality, and reducing educational inequality, and 
systematically investigate differences in use, perceptions, and capabilities in using 
computer technologies by student gender. 


Seamus Hegarty 
Leslie Rutkowski 
Series editors 
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Chapter 1 
Socioeconomic Inequality and Educational 55 
Outcomes: An Introduction 


Abstract International large-scale assessments may be used to understand differences 
in average student performance across countries over time. Differences in achievement 
are associated with students’ background characteristics and, while the most salient 
background variables may differ across education systems, a substantial amount of the 
variance in student achievement is normally explained by family socioeconomic status 
(SES). Family SES is thus considered an important factor in education research, but 
there are still challenges regarding how to best measure SES operationally. In addition 
to measuring student achievement, the IEA’s Trends in International Mathematics and 
Science Study (TIMSS) has been collecting background information from students, 
their teachers, and principals at four-year intervals since 1995. This study uses data 
from the IEA’s TIMSS grade eight mathematics and science assessments to examine 
changes in the achievement gaps between high- and low-SES students between 1995 
and 2015 for 13 education systems, as operationalized by a modified version of the 
TIMSS home educational resources index. The performance of disadvantaged students 
may also be tracked over time. These measures explain a sizeable amount of variation 
in the TIMSS achievement scores and, in conjunction with data on the countries’ 
education systems and other macroeconomic indicators, can provide country-specific 
analyses and an across-country synthesis. 


Keywords Achievement gaps - Educational inequality - International large-scale 
assessment - Socioeconomic status (SES) - Student achievement - Trend analysis - 
Trends in International Mathematics and Science Study (TIMSS) - Australia - 
Hong Kong - Hungary - Islamic Republic of Iran - Lithuania - New Zealand - 
Norway - Republic of Korea - Russian Federation - Singapore - Slovenia - 
Sweden - United States 
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Educational inequality is a central theme in education research due to its strong 
association with the attainment of educational qualifications and social positions in a 
society, which more importantly mirrors inequality in a society (Breen and Jonsson 
2005). Ferreira and Gignoux (2011) described the difference between two types of 
inequality (as well as methods for measuring them): (1) inequality of outcomes, 
which can be expressed simply as the degree of variability in the outcome measure, 
and (2) inequality in opportunity, which can be described as the degree to which 
family background and other pre-determined personal characteristics determine a 
person's educational outcomes (including, for example, differential access to better 
resourced schools). Our study focuses on the second type of inequality, and 
specifically on the degree to which family socioeconomic status (SES) is related to 
educational outcomes as measured in the Trends in International Mathematics and 
Science Study (TIMSS). When we write about inequality or educational inequality 
further on in the report, especially in the results section, it should be interpreted as 
"educational inequality in opportunity due to SES." Researchers have made 
substantial contributions to the understanding of how SES relates to student 
performance (Lee and Burkam 2002). Sociological theories have been proposed 
and tested regarding how economic, social, and cultural capital in the family of 
origin can help transmit advantages from one generation to the next (Bourdieu 1986; 
Coleman 1988, 1990; Lareau 2011). 

Over the last few decades, the emergence of international comparative assessment 
has provided an opportunity for researchers and policymakers to not only track 
student performance within their own education system but also to compare their 
system's performance with that of other participating education systems. Prior to 
this, researching the inequality of educational outcomes depended on using different 
data sets from the various education systems to make comparisons across societies 
(Shavit and Blossfeld 1993). However, student performance results from 
international comparative assessments, such as IEA's TIMSS and Progress in 
International Reading Literacy Study (PIRLS), and the Program for International 
Student Assessment (PISA) run by the Organisation for Economic Cooperation and 
Developemt (OECD), are followed today not only by researchers and policymakers 
but also by an informed public that is increasingly interested in how the results of 
these assessments may be used to understand how well each education system 
educates its students. For example, in many countries, the mainstream media has 
been closely following the international assessment results and making comparisons 
between students in their own country with their international counterparts 
(Coughlan 2016; Strauss 2017). As another example, the success of students in 
Asian educational systems is attributed by many media outlets to their coherent 
approach and having a culture devoted to education (Coughlan 2016). 

In addition to tracking and trying to understand international differences in 
average student performance as they evolve over time, public debate also centers 
on the dispersion in achievement and the impact of social background on educational 
achievement within each education system. For instance, grade eight students in the 
United States have significantly improved their mathematics achievement over the 
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past 20 years according to results from TIMSS. However, in the United States, the 
gaps between students at the high- and low-ends of the achievement distribution 
have widened over time (Sparks 2016). 

Ideally, any achievement gaps between students should reflect only differences in 
their abilities or efforts. However, in most education systems, strong achievement 
differences exist that are associated with students’ background characteristics, such 
as race/ethnicity, gender, rural/urban residence status, or immigration/migration 
status. Depending on a society's historic and social context, the most salient 
background variables may differ across education systems. Yet one background 
variable explaining a significant amount of variance in students’ achievement scores 
is common among countries, namely SES background, making this one of the most 
critical variables in education research. 

While the association between SES and students’ academic achievement has been 
well documented in the literature, research on how such associations have become 
stronger or weaker over time in individual education systems is limited. Most 
national education policies include an aim to narrow the SES achievement gap and 
to improve performance among disadvantaged students. However, quantifiable 
measures of these efforts are still lacking, especially those that are straightforward 
and easy to understand. 

TIMSS has assessed student academic performance and collected various kinds 
of background information from students, their teachers, and principals at four-year 
intervals since 1995. TIMSS data sets spanning two decades allow for the 
exploration of many interesting research questions, but, in this report, we will 
focus on two research questions concerning the changes over time in SES 
achievement gaps and in the performance of disadvantaged students, within and 
between education systems. For both, we identified potential indicators within the 
TIMSS data sets. 


1. How has the inequality of education outcomes due to family socioeconomic status 
changed for different education systems between 1995 and 2015? 


* Indicator: Change over time in the difference in TIMSS scores between the 
lowest and highest SES quartiles within each education system. 


2. To what extent have education systems managed to increase the academic 
performance of disadvantaged students between 1995 and 2015? 


* Indicator: Change over time in the percentage at or above the TIMSS 
intermediate benchmark for the lowest SES quartile in each education system. 


To address our research questions, we pooled the data for all cycles of the TIMSS 
grade eight mathematics and science assessments and examined the described trend 
measures for the education systems that had at least three data points over the 
20 years of TIMSS (education systems that had participated in 1995, 2015, and in 
at least one intermediate cycle). We identified 13 educational systems that satisfied 
this criterion, namely Australia, Hong Kong, Hungary, Islamic Republic of Iran, 
Lithuania, New Zealand, Norway, Republic of Korea, Russian Federation, 
Singapore, Slovenia, Sweden, and the United States. 
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Over time, a consensus has been reached that SES is comprised of (at a minimum) 
parental education, family income, and parents’ occupation (or occupational 
prestige) since these three indicators reflect different aspects of family background 
(Brese and Mirazchiyski 2013). Among empirical studies, however, there are still 
challenges on how to best operationally measure SES (Buchmann 2002). This is 
especially true for large-scale assessments, where collection of all three components 
from objective sources is simply not feasible due to privacy concerns and funding 
constraints. In addition, there are concerns about information accuracy, especially 
when students are asked about family income and parental occupation. Most large- 
scale assessments thus ask students to report only on their parents' education levels 
(and even that is normally not done for students below lower secondary school age). 
Parental education, conceptually, plays a key role among the three SES components, 
as educational attainment, to some extent, functions as a gatekeeper for obtaining 
certain types of occupations that carry higher occupational prestige and are 
associated with higher income. 

These practical considerations therefore leave some room for researchers to 
decide what additional proxy variables to use as SES measures. Yang (2003) 
confirmed in their study that possession of a set of household items may be used 
as an SES indicator. Another potential indicator is *number of books at home," and 
unpublished research by Ogut et al. (2016)! has suggested that this variable, as 
reported by students, is strongly related to the level of parental education reported 
by their parents. All in all, even with the variability and limitations of the 
measurement of SES, one important criterion to judge the effectiveness of an 
SES measure is the extent of its association with students' performance (Broer 
et al. unpublished work 2017), as long as all the components of the SES measure 
are conceptually related to SES. 

This study introduces a modified version of the TIMSS home educational 
resources (HER) index, which is comparable over the 20-year period of TIMSS 
data. We argue that the measure constructed for this study, referred to in this book as 
the SES* index’, is based on educationally relevant aspects of the SES construct, and 
that it is an effective measure, explaining a sizeable amount of variation in TIMSS 
achievement scores. 


ІВ. Оги, G. Bohrnstedt, & M. Broer presented a paper entitled “Investigating SES using the 
NAEP-HSLS overlap sample” at the United States’ National Council on Measurement in Education 
(NCME) 2016 annual meeting in Washington, DC, in April 2016. Copies of the presentation are 
available from M. Broer by sending a request to mbroer @air.org. 

2М. Broer, О. Xie, & G.W. Bohrnstedt presented a paper entitled “Construction and validation of a 
new proxy socioeconomic status measure for NAEP grade 12" during the Development and 
Applications of Alternative Student Socioeconomic Status Measures session of the American 
Educational Research Association (AERA) 2017 conference in San Antonio, TX, in April 2017. 
Copies of the presentation are available from M. Broer by sending a request to mbroer@air.org. 
3Since this index does not represent the full SES construct, as usually defined by parental education, 
family income, and parental occupation, we used an asterisk (SES*) to denote the conceptual 
difference. 
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In the remainder of this book, we comprehensively review the existing literature 
on socioeconomic status and educational achievement, and, based on this and our 
own research, propose a robust new comparative socioeconomic index measure for 
the TIMSS data. We applied this new methodology to analyze TIMSS trend data to 
describe how socioeconomic achievement gaps have evolved over time for the 
studied education systems. Finally, we discuss these gaps from a macroeconomic 
perspective, and identify opportunities for future research. An appendix provides a 
sensitivity check of the results, better establishing the validity of our SES* index. 
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Chapter 2 A 
A Review of the Literature a 
on Socioeconomic Status and Educational 
Achievement 


Abstract The foundations of socioeconomic inequities and the educational 
outcomes of efforts to reduce gaps in socioeconomic status are of great interest to 
researchers around the world, and narrowing the achievement gap is a common goal 
for most education systems. This review of the literature focuses on socioeconomic 
status (SES) and its related constructs, the association between SES and educational 
achievement, and differences among educational systems, together with changes 
over time. Commonly-used proxy variables for SES in education research are 
identified and evaluated, as are the relevant components collected in IEA's Trends 
in International Mathematics and Science Study (TIMSS). Although the literature 
always presents a positive association between family SES and student achievement, 
the magnitude of this relationship is contingent on varying social contexts and 
education systems. TIMSS data can be used to assess the magnitude of such 
relationships across countries and explore them over time. Finally, the literature 
review focuses on two systematic and fundamental macro-level features: the extent 
of homogeneity between schools, and the degree of centralization of education 
standards and norms in a society. 


Keywords Centralization versus decentralization - Educational inequality - Forms 
of capital - Homogeneity versus heterogeneity - International large-scale assessment - 
Student achievement - Socioeconomic status - Trends in International Mathematics 
and Science Study (TIMSS) 


Educational inequality occurs in multiple forms. Van de Wefhorst and Mijs (2010) 
discussed its existence through the inequality of educational opportunity in terms of 
the influence of social background on students' test scores, as well as in learning, as 
expressed by the performance distribution in test scores. According to the authors, 
these two characteristics of inequality are conceptually different in that an 
educational system may have equality in terms of dispersion (or variance) in 
educational achievement but inequality in terms of opportunities; yet, in general, 
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societies that are equal in terms of dispersion are also more equal in terms of 
opportunities. 

Different education systems take part in each cycle of TIMSS, but 25 education 
systems took part in the grade eight mathematics student assessment in both 1995 
and 2015. For these 25 participating systems, the average mathematics achievement 
score increased by only five score points between 1995 and 2015 (Mullis et al. 
2016). Focusing only on more recent trends, for the 32 education systems that 
participated in the grade eight mathematics student assessment in both 2011 and 
2015, there was a gain of nine scale score points between 2011 and 2015, suggesting 
that many of the education systems with the largest gains are those starting from a 
low base. As there is limited information on family and home background and its 
relationship with TIMSS international achievement, this spread in achievement is 
not sufficient to explain why education systems perform differently. Therefore, our 
study focuses on the other aspect of educational inequality, namely how SES 
background is related to educational achievement. In the next two sections of this 
chapter, we review the concept and measurement of socioeconomic status, and the 
literature regarding the relationship between family SES and student academic 
achievement. The rest of this chapter focuses on differences between the various 
education systems and changes in educational inequality over time. 


2.1 Socioeconomic Status and Related Constructs 
and Measures 


The American Psychological Association (APA) defines socioeconomic status as 
"the social standing or class of an individual or group" (APA 2018). SES has been 
commonly used as a latent construct for measuring family background (Bofah and 
Hannula 2017). However, among empirical studies, there is no consensus on how to 
best operationalize the concept. In many studies, the measurement of SES does not 
receive much attention, with very limited discussion over why certain indicators 
were used rather than others (Bornstein and Bradley 2014). Liberatos et al. (1988) 
argued that there was no one best measure, because the choice of the SES measure 
depended on the conceptual relevance, the possible role of social class in the study, 
the applicability of the measure to the specific populations being studied, the 
relevance of a measure at the time of study, the reliability and validity of the 
measure, the number of indicators included, the level of measurement, the 
simplicity of the measure, and comparability with measures used in other studies. 
Historically, SES has been conceptualized and measured in various ways. 
Taussig (1920) conceptualized SES as the occupational status of the father. Later, 
Cuff (1934) adopted a score card proposed by Sims (1927) as a measure of SES; this 
included questions about items possessed by the home, parents' education, father's 
occupation, and other relevant information. Moving on from these early studies, 
development of instruments for measuring SES has become more complicated, 
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including more advanced methods such as factor analysis or model-based 
approaches (NCES [National Center for Educational Statistics] 2012). By the 
1980s, one general agreement had emerged: SES should be a composite variable, 
typically measuring education, income, and occupation, since these three indicators 
reflect different aspects of family background (Brese and Mirazchiyski 2013). 

However, collecting this information is known to be challenging. Besides privacy 
concerns, there are also concerns about information accuracy (Keeves and Saha 1992). 
For example, the National Assessment of Educational Progress (NAEP) in the United 
States does not collect family income or parental occupation directly from students, as 
many of them are unable to accurately report such data (Musu-Gillette 2016). 
Similarly, TIMSS decided not to include questions about parental occupation and 
income because of doubts about the reliability and utility of similar information 
collected by previous IEA surveys (Buchmann 2002). Therefore, the grade eight 
student questionnaires for TIMSS include only three proxy components for SES: 
parental education, books at home, and home possessions (such as ownership of a 
calculator, computer, study desk, or dictionary), with some evolution in the home 
possession items over time owing to rapid advancements in technology over the 
20 years of TIMSS (more recent items include the internet, or computer tablet, for 
example). 

The abstract nature of the concept of SES leaves some room for researchers to 
decide what proxy variables to use as SES measures. Yang (2003), for example, 
found that the possession of a set of household items may be used as SES 
indicators. Despite variability and limitations in the measurement of SES, its 
association with student performance has been demonstrated in numerous studies 
(Sirin 2005). 


2.2 Family SES and Student Achievement 


Theoretical and empirical work has emphasized that family SES has an impact on 
children's educational outcomes, examined mechanisms through which family SES 
is related to children's achievement, and identified potential pathways behind this 
relationship, one of which uses three forms of capital: economic, cultural, and social 
capital (Bourdieu 1986; Coleman 1988, 1990). In other words, differences in the 
availability of these forms of capital! across households eventually lead to disparities 
in children's academic achievement (Buchmann 2002). 

Bourdieu (1986) posited that capital can present itself in three fundamental 
forms and that economic capital is the source of all other forms of capital. The 
other types of capital are treated as transformed and disguised forms of economic 
capital. Economic capital can be used in pursuit of other forms of capital; for 


'Note that family socioeconomic status is clearly related to Bourdieu's theory of capital in the 
empirical world. Conceptually, however, they do not equate with each other. 
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example, family income can be used to pay for organized after-school activities, to 
access elite educational opportunities, or to build up valuable social networks 
(Lareau 2011). Children from disadvantaged backgrounds are constrained by the 
financial resources they and their family possess (Crosnoe and Cooper 2010). As 
such, economic capital determines the extent to which parents can offer financial 
support to children's academic pursuits. 

In addition to economic capital, cultural capital, namely knowledge of cultural 
symbols and ability to decode cultural messages, helps parents transmit their 
advantages to children and to reproduce social class (Bourdieu 1986). According 
to Bourdieu (1986), an individual's cultural capital can exist in an embodied state as 
well as in an objectified state. In the embodied state, cultural capital focuses on 
"physical capital" where the body itself is a marker of social class, as particular 
embodied properties exist as a consequence of specific class practices (Tittenbrun 
2016). Through this state, inequality in socioeconomic class can find expression in 
embodied ways, such as physical appearance, body language, diet, pronunciation, 
and handwriting. In the objectified state, inequality is expressed in forms of cultural 
goods, such as accessibility to pictures, books, dictionaries, and machines. 
Therefore, in this view, Bourdieu sees the body and cultural goods as forms of 
currency that result in the unequal accumulation of material resources and, by 
extension, represent an important contributor to class inequality (Perks 2012). 

Children from higher social classes also have advantages in gaining educational 
credentials due to their families. Cultural capital is considered an important factor for 
school success. Yang (2003) suggested possession of cultural resources had the most 
significant impact on students' mathematics and science achievement in most 
countries. If cultural resources are differentiated according to family background, 
and if some cultural resources have more value than others in the education system, it 
is reasonable to assume that differential achievement is related to an individual's 
social class (Barone 2006). For example, a student's social ability and language 
style, as well as attitudes toward the school curriculum and teachers, may differ 
according to social class origins (Barone 2006). As such, parental school choice in 
some countries favors children from those families that already possess dominant 
cultural advantages (1.е., children attending private schools in the United States), 
thus confirming the cultural inequalities between classes and status groups of 
families to produce educational inequalities among their children (Shavit and 
Blossfeld 1993). Lareau (1987, 2011) further posited that middle-class parents 
have a different parenting style, which she termed concerted cultivation, fostering 
their child's talent through organized activities, while working-class parents tend to 
have a natural growth parenting style, letting their children create their own activities 
with more unstructured time. Consequently, middle-class families prepare their 
children better for school since their parenting style is more valued and rewarded 
by the school system. 

Finally, the possession of social capital reflects the resources contained in social 
relations, which can be invested with expected benefits (Bourdieu 1986). 
Differences in educational success can be attributed to different levels of existing 
social capital, which is produced in networks and connections of families that the 
school serves (Rogošić and Baranović 2016). Coleman (1988) developed а 
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conceptual framework of social capital in which social structure can create social 
capital, through family, school, and community. The relationships between the 
family and the community may be used to explain the higher educational 
achievements of students based on expected achievements with respect to their 
socioeconomic status (Mikiewicz et al. 2011). 

In summary, while the overall association between family SES and students’ 
academic achievement is well documented in theoretical and empirical work, the 
magnitude of the relationship between family SES and achievement differs across 
countries. This may be related to differences in education systems and jurisdictions, 
and societal changes over time. 


2.3 Differences in Education Systems and Changes 
Over Time 


In any society, there are two systematic and fundamental macro-level features that 
highlight the differences in education systems and how they have changed over time. 
First, is the extent of homogeneity among education systems. Second, is the degree 
of centralization of education standards and norms in a society. The association 
between family background and children's achievement depends on the education 
system and the social context (i.e., the level of homogeneity and centralization). 
Where educational inequality is prominent, students from different backgrounds 
may demonstrate larger achievement gaps. 


2.3.1 Homogeneous Versus Heterogeneous 


Previous research has shown that students at lower levels of SES perform better in 
education systems with lower levels of inequality than their counterparts in countries 
with more significant SES differences (Ornstein 2010). That is, some education 
systems are more homogeneous than others, with schools being more similar to each 
other in terms of funding. As an example, Finnish households have a narrow 
distribution of economic and social status at the population level and their schools 
show little variation in terms of funding (Mostafa 2011). 

Furthermore, Mostafa (2011) found that school homogeneity on a large scale is a 
source of equality since it diminishes the impact of school characteristics on 
performance scores. Finland is often seen as an example of a homogeneous 
education system with high levels of similarity between schools, which in turn 
reduces the impact of school variables on performance scores (Kell and Kell 2010; 
Mostafa 2011). More specifically, Montt (2011) examined more than 50 school 
systems, including Finland, in the 2006 cycle of PISA and found that greater 
homogeneity in teacher quality decreased variability in opportunities to learn 
within school systems, potentially mitigating educational inequality in achievement. 
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By contrast, Hong Kong has a relatively high-income disparity compared to other 
societies (Hong Kong Economy 2010). However, the relationship between 
socioeconomic status and mathematics achievement was found to be the lowest 
among the education systems participating in the 2012 cycle of PISA (Ho 2010; 
Kalaycioglu 2015). This suggests that, despite diversity in their SES background, 
most students from Hong Kong access and benefit from the education system 
equally. Hong Kong's high performance in reading, mathematics, and science also 
suggests the average basic education is of high quality (Ho 2010). 

However, in many other countries with heterogeneous education systems, 
educational inequality has manifested itself primarily through the stratification of 
schools on the basis of socioeconomic composition, resource allocation, or locale. 
For example, unlike schooling in many other countries, public schooling policies in 
the United States are highly localized. Local property taxes partially finance public 
schools, school assignments for students depend on their local residence, and 
neighborhoods are often divided by racial and socioeconomic background 
(Echenique et al. 2006; Iceland and Wilkes 2006). Cheema and Galluzzo (2013) 
confirmed the persistence of gender, racial, and socioeconomic gaps in mathematics 
achievement in the United States using PISA data from its 2003 cycle. Inequalities in 
children's academic outcomes in the United States are substantial, as children begin 
school on unequal terms and differences accumulate as they get older (Lareau 2011; 
Lee and Burkam 2002). 

In Lithuania, there has also been a growing awareness that an ineffectively 
organized or poorly functioning system of formal youth education increases the 
social and economic divide and the social exclusion of certain groups (Gudynas 
2003). To ensure the accessibility and quality of educational services in Lithuania, 
special attention has traditionally been paid to a student's residential location. 
Gudynas (2003) suggested that the achievement of pupils in rural schools in 
Lithuania was lower than that of pupils in urban schools, with the difference being 
largely explained by the level of parental education in rural areas, which was on 
average lower than that of urban parents. Similarly, in New Zealand, residential 
location is considered to be a barrier to educational equality. Kennedy (2015) 
observed that students residing in rural residential areas on average tend to have 
lower SES than those in urban areas, and receive a considerably shorter education 
than their counterparts living in urban centers, thereby promoting SES disparities in 
access to education. 

In the Russian Federation, Kliucharev and Kofanova (2005) noted that the 
inequality between well-off and low-income individuals regarding access to 
education has been increasing since the turn of the century. According to 
Kosaretsky et al. (2016), the greatest inequality in educational access in the 
Russian Federation was observed in the 1990s, where the rising number of 
educational inequalities was largely determined by the accelerating socioeconomic 
stratification of the population, as well as significant budget cuts to education. 
Although the state articulated policies aiming for universal equality of educational 
opportunities, they argued that the policies were not implemented with the required 
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financial and organizational support. As a result, in the immediate post-Soviet era, 
the Russian Federation has observed increasing educational inequality and some loss 
of achievement compared to the Soviet period. 

A final example is Hungary. Horn et al. (2006) noted that OECD’s PISA studies 
in the early 2000s highlighted the need for the Hungarian school system to improve 
both in effectiveness and equality. They contended that achievement gaps among 
schools make the Hungarian education system one of the most unequal among the 
participating countries in the PISA 2000 and 2003 cycles. The variation in 
performance between schools in Hungary is alarmingly large, about twice the 
OECD average between-school variance (OECD 2004). By contrast, the within- 
school variance is less pronounced, suggesting that students tend to be grouped in 
schools with others sharing similar characteristics. In other words, students' 
achievement gaps seemingly mirror the differences in socioeconomic backgrounds 
of students across different schools (OECD 2001, 2004). In recent years, persistent 
education performance gaps with regard to socioeconomic background of students 
have been observed in Hungary, with 23% of the variation in students’ mathematics 
performance being explained by differences in their SES background, well above the 
average of 15% for OECD countries (OECD 2015). 


2.3.2 Centralized Versus Decentralized 


In addition to differences in homogeneity, education systems can be classified as 
centralized or decentralized. A centralized education system is one that would have 
centralized education funding (e.g., at the national level) across the education system 
with little local autonomy, while in decentralized education systems, municipalities 
oversee school funding for both public and private schools (Bóhlmark and Lindahl 
2008; Oppedisano and Turati 2015). Centralization generally leads to the 
standardization of curriculum, instruction, and central examinations in an education 
system, and can be helpful in reducing inequalities since it mitigates the influence of a 
student's family background (Van de Wefhorst and Mijs 2010). By contrast, high 
levels of decentralization can create greater disparities between schools, especially 
when the level of funding is determined by the local context (Mostafa 2011). 

Sweden is an example of a decentralized education system that was centralized 
until the implementation of wide-reaching reforms in the early 1990s (Hansen et al. 
2011). The previously centralized Swedish school system has been thoroughly 
transformed into a highly decentralized and deregulated one, with a growing 
number of independent schools and parental autonomy in school choice 
(Bjórklund et al. 2005). Concurrently, examining multi-level effects of SES on 
reading achievement using data from IEA's Reading Literacy Study from 1991 and 
PIRLS data from 1991 to 2001, the SES effect appears to have increased in Sweden 
over time, with between-school differences being greater in 2001 than in 1991, 
suggesting school SES has a strong effect (Hansen et al. 2011). 

Similarly, there has also been growing debate about educational inequality in the 
Republic of Korea in recent years. By analyzing grade eight TIMSS data from the 
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1999, 2003, and 2007 cycles of the assessment, Byun and Kim (2010) found the 
contribution of SES background on student achievement had increased over time. 
They suspected the higher educational inequality might be related to various factors, 
including a widening income gap and recent educational reforms geared toward 
school choice, as well as increased streaming by academic ability and curriculum 
differentiation created by a decentralized education system. 

Researchers have found evidence to support the view that decentralized education 
systems in developed countries perform better than centralized systems in terms of 
reducing students’ achievement inequality (see, e.g., Rodríguez-Pose and Ezcurra 
2010). Conversely, Causa and Chapuis (2009) used PISA data for the OECD 
countries to confirm that decentralized school systems were positively associated 
with equity in educational achievement. Furthermore, according to PISA 2000 and 
2006, in European countries inequality in educational outcomes has apparently 
declined in decentralized school systems, while it has concomitantly increased in 
centralized systems (Oppedisano and Turati 2015). 

Mullis et al. (2016) argued that efficiency and equality can work together. They 
found that many countries have improved their TIMSS national averages while also 
reducing the achievement gap between low- and high-performing students. 
Similarly, an analysis using TIMSS scores from 1999 and 2007 discovered a 
prominent inverse relation between the within-country dispersion of scores and the 
average TIMSS performance by country (Freeman et al. 2010; Mullis et al. 2016). 
The pursuit of educational equality does not have to be attained at the expense of 
equity and efficiency. 

In conclusion, the positive association between family background and children's 
achievement is universal. However, the magnitude of such associations depend on 
the social context and education system. In other words, the achievement gap 
between students from different backgrounds is more pronounced in education 
systems where overall inequality (e.g., income inequality) is strong. Narrowing the 
achievement gap is a common goal for most education systems. But it is well 
understood that stagnant scores for low-SES students and declines in the scores of 
high-SES students should not be seen as an avenue for enhancing equality. Rather, 
education systems should strive for equality by improving the performance of all 
students while focusing on improving the achievement of low-SES students at a 
faster rate to reduce gaps in achievement (Mullis et al. 2016). In recognition of this, 
our study not only focuses on how inequalities in educational outcomes relate to 
socioeconomic status over time for select participating education systems in TIMSS 
but also tracks the performance of low-SES** students separately. In order to make а 
comparable trend analysis, we first constructed a consistent measure of family SES* 
based on a modified version of the TIMSS HER. Chapter 3 describes the data and 


?The SES measure used in this study is a modified version of the TIMSS home educational 
resources (HER) index and does not represent the full SES construct, as usually defined by parental 
education, family income, and parental occupation. In this report, we therefore term our measure 
SES* to denote the conceptual difference (Please refer to Chap. 1 for more details). 
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methods used in the study and Chap. 4 presents the trends in SES* achievement gaps 
of the 13 education systems that participated in three cycles of TIMSS, including the 
1995 and 2015 cycles. 
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Chapter 3 N) 
Methodology: Constructing ic 
a Socioeconomic Index for TIMSS Trend 
Analyses 


Abstract To assess education system trends in the inequality of educational 
outcomes, a robust socioeconomic index for TIMSS trend analysis is needed. This 
chapter begins by outlining the TIMSS data and sample design, as well as changes in 
the sample design over time, with special emphasis on the aspects that are 
specifically related to family socioeconomic status and student achievement. To 
analyze trends over the entire 20 years of TIMSS data, the analysis was limited 
to education systems that participated in the first cycle in 1995, the most recent 
cycle in 2015, and at least one other administration in between. After assessing 
the completeness of the data, 13 educational systems were included in the study: 
Australia, Hong Kong, Hungary, Islamic Republic of Iran, Lithuania, New Zealand, 
Norway, Republic of Korea, Russian Federation, Singapore, Slovenia, Sweden, and 
the United States. Items used for constructing the SES index were the number of 
books at home, home possessions, and the highest level of education of either parent. 
Students in each educational system were grouped into high- and low-SES groups 
based on the SES distribution for a given year. Constructing a consistent measure of 
SES across all TIMSS cycles is an important contribution to research that uses 
TIMSS for trend analyses. In addition to analyzing the achievement gaps over 
time, examining trends in performance among low-SES students in each education 
system provides additional information on how education systems are addressing the 
issues facing disadvantaged students. 


Keywords Index construction - International large-scale assessment - Measures of 
educational inequality - Multiple imputation - Plausible values - Socioeconomic 
status (SES) - Trends in International Mathematics and Science Study (TIMSS) 
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3.1 TIMSS Data and Sample Characteristics 


We used the TIMSS grade eight public-use data from 1995 through 2015 to establish 
how inequalities of education outcomes have changed between 1995 and 2015, and 
to assess whether education systems have managed to increase the performance of 
disadvantaged students. TIMSS has been conducted every four years since 1995 to 
monitor trends in mathematics and science performance of students across education 
systems. Every participating education system provides a representative sample of 
students by adopting a two-stage random sample design. Typically, in each 
participating education system, a sample of schools is drawn at the first stage, and 
one or more intact classes of students from each of the sampled schools are selected 
at the second stage (LaRoche et al. 2016). Although most features have remained 
constant over the different TIMSS cycles, there have also been several significant 
changes in sample design, country participation, and questionnaire administration. 

First, the target population has changed slightly. The first cycle of TIMSS in 1995 
identified three target populations; one of them was students enrolled in the two 
adjacent grades, which maximized coverage of 13-year-olds (Foy et al. 1996). At the 
time of testing, most students were either in the grade seven or grade eight. This 
practice was refined for the 1999 cycle of TIMSS, and resulted in only grade eight 
students being assessed. To maintain comparability, for our study, we therefore 
only included grade eight students for most education systems in the 1995 
assessment in our trend analyses, which is in alignment with the practice outlined 
in the 77MSS 1999 international mathematics report (Mullis et al. 2000) and TIMSS 
2015 international results in mathematics (Mullis et al. 2016, appendix A.1, at 
http://timssandpirls.bc.edu/timss2015/international-results/timss-2015/mathematics/ 
appendices/).! Norway was the only exception, because Norway only included grade 
six and seven students in its 1995 sample. However, according to the TIMSS 2015 
report, the sample of upper-grade students (grade seven) in Norway in 1995 was 
comparable to that in 2015 (see Mullis et al. 2016, appendix A.1). Therefore, in the 
case of Norway, we kept the sample of grade seven students in 1995 for trend 
comparison (Gonzalez and Miles 2001)? 

Second, although many education systems have participated in TIMSS over the 
last 20 years, not every education system participated in each cycle. To analyze 
trends over the entire 20 years of TIMSS data, we therefore limited our analysis to 
those education systems that participated in the first cycle in 1995, the most recent 


‘In the TIMSS 1999 international mathematics report Mullis et al. (2000) examined trends іп 
mathematics achievement between 1995 and 1999. The 1995 average scale score was calculated for 
grade eight students only in Exhibit 1.3 (see pp. 34—36). 

? According to the TIMSS 1999 user guide for the international database, the TIMSS 1999 target 
grade was the upper grade of the TIMSS 1995 population 2 and was expected to be the grade eight 
in most countries. However, for Norway, it was the seventh grade (see Exhibit 5.2 Grades tested in 
TIMSS 1995-Population 2). Please refer to https://timss.bc.edu/timss1999i/data/bm2, userguide. 
pdf 
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cycle in 2015, and at least one other intermediate administration cycle. This 
produced a potential sample of 18 education systems." 

However, according to the 2015 TIMSS international results in mathematics 
(see Mullis et al. 2016, appendix A.1), many education systems’ previous data 
cannot be used for trend analysis to 2015. This is primarily due to improved 
translations or increased population coverage. For example, the data for Australia 
in 1999, Kuwait in 1995 and 2007, Canada in 1995 and 1999, Israel in 1995, 1999, 
2003, and 2007, Slovenia in 1999, and Thailand in 1995 were not considered 
comparable to 2015 data. Therefore, four education systems (Canada, Israel, 
Kuwait, and Thailand) had to be excluded from the analyses because 1995 data 
cannot be used for trend analyses. 

In addition, given that our primary focus is SES-related information, we excluded 
England from our study since it did not have data for parental education in 1995, 
1999, and 2007. In total, our analytical sample is limited to the following 
13 education systems (Table 3.1): 


* Australia, Hong Kong, Hungary, Islamic Republic of Iran, Lithuania, Republic of 
Korea, Russian Federation, Singapore, Slovenia, and the United States (education 
systems that participated in all six cycles); and 

* New Zealand (which participated in 1995, 1999, 2003, 2011, and 2015), Norway 
(which participated in 1995, 2003, 2007, 2011, and 2015), and Sweden (which 
participated in 1995, 2003, 2007, 2011, and 2015). 


Finally, for trend analysis, several adjustments were made to follow the approach 
used by Mullis et al. (2016). First, IEA has a policy that students should not fall 
under the minimum average age of 13.5 years (for grade eight) at the time of testing 
(see Mullis et al. 2016, appendix C.10). Therefore, New Zealand assessed students in 
grade nine across multiple cycles. The results for grade nine students in 1995, 1999, 
2011, and 2015 are deemed comparable to those for grade eight students who 
participated in 2003 in New Zealand. Second, although Slovenia assessed grade 
eight students in 1995, the results for grade eight students in 1995 are not deemed 
comparable to those in other cycles. Therefore, data for grade seven students in 1995 
is used for trend analysis. Third, in Lithuania, the results for students assessed in 
Polish or Russian in 2015 are deemed not comparable to previous cycles. Therefore, 
trend results only include students assessed in Lithuanian and do not include 
students assessed in Polish or Russian in 2015. 


>The potential sample included (1) 11 education systems that participated in all six cycles: 
Australia, Hong Kong, Hungary, Islamic Republic of Iran, Israel, Republic of Korea, Lithuania, 
Russian Federation, Singapore, Slovenia, the United States, and (2) seven education systems that 
participated in both 1995 and 2015 and in at least one other administration: England, New Zealand, 
Norway, Sweden, Thailand, Canada, and Kuwait. 
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Table 3.1 Samples for each education system in each TIMSS assessment year 
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Education Sample characteristics TIMSS cycle 

system 1995 1999 | 2003 2007 | 2011 2015 

Australia Overall student sample 12,852 | 4032 | 4791 4069 | 7556 10,338 
Grade level(s) used for | G8 n/c G8 G8 G8 G8 
trend analysis 
Number of students in 7392 n/c 4791 4069 | 7556 10,338 
trend sample 

Hong Kong Overall student sample | 6752 5179 | 4972 3470 | 4015 4155 
Grade level(s) used for | G8 G8 G8 G8 G8 G8 
trend analysis 
Number of students in 3339 5179 | 4972 3470 | 4015 4155 
trend sample 

Hungary Overall student sample | 5978 3183 | 3302 4111 | 5178 4893 
Grade level(s) used for | G8 G8 G8 G8 G8 G8 
trend analysis 
Number of students in 2912 3183 | 3302 4111 | 5178 4893 
trend sample 

Islamic Overall student sample | 7429 5301 | 4942 3981 | 6029 6130 

Republic of Grade level(s) used for | G8 G8 G8 G8 G8 G8 

Iran trend analysis 
Number of students in 3694 5301 | 4942 3981 | 6029 6130 
trend sample 

Lithuania Overall student sample | 5056 2361 | 4964 3991 | 4747 4347 
Grade level(s) used for | G8 G9 G8 G8 G8 G8 
trend analysis 
Number of students in 2525 2361 | 4964 3991 |4747 2933 
trend sample 

New Zealand Overall student sample | 6867 3613 |3801 n/a 5336 8142 
Grade level(s) used for | СО G9 G8 n/a G9 G9 
trend analysis 
Number of students in 3683 3613 |3801 n/a 5336 8142 
trend sample 

Norway Overall student sample | 5736 n/a 4133 4627 |3862 4795 
Grade level(s) used for | G7 n/a G8 G8 G8 G8 
trend analysis 
Number of students in 3267 n/a 4133 4627 |3862 4795 
trend sample 

Republic of Overall student sample | 5827 6114 | 5309 4240 |5166 5309 

Korea Grade level(s) used for | G8 G8 G8 G8 G8 G8 
trend analysis 
Number of students in 2920 6114 | 5309 4240 |5166 5309 
trend sample 

Russian Overall student sample | 8160 4332 | 4667 4472 | 4893 4780 

Federation Grade level(s) used for | G8 G8 G8 G8 G8 G8 
trend analysis 
Number of students in 4022 4332 | 4667 4472 |4893 4780 


trend sample 


(continued) 
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Table 3.1 (continued) 


Education Sample characteristics TIMSS cycle 
system 1995 1999 | 2003 2007 |2011 2015 
Singapore Overall student sample | 8285 4966 |6018 4599 |5927 6116 


Grade level(s) used for | G8 G8 G8 G8 G8 
trend analysis 


Number of students in 4644 4966 | 6018 4599 |5927 
trend sample 


G8 


6116 


Slovenia Overall student sample | 5606 3109 |3578 4043 |4415 4257 
Grade level(s) used for | G7 n/c G7 G8 G8 G8 
trend analysis and 

G8 


Number of students in 2898 n/c 3578 4043 |4415 
trend sample 
Sweden Overall student sample | 8855 n/a 4256 5215 | 5573 
Grade level(s) used for | G8 n/a G8 G8 G8 
trend analysis 
Number of students in 1949 n/a 4256 5215 | 5573 
trend sample 
United States Overall student sample 10,973 | 9072 | 8912 7377 |10,477 | 10,221 
Grade level(s) used for | G8 G8 G8 G8 G8 G8 
trend analysis 


Number of students in 7087 9072 | 8912 7377 |10,477 | 10,221 
trend sample 


4257 


4090 
G8 


4090 


Source International Association for the Evaluation of Educational Achievement (IEA) Trends in 
International Mathematics and Science Study (TIMSS) 1995, 1999, 2003, 2007, 2011, and 2015 
Mathematics and Science Assessments (see www.iea.nl/data) 

Notes G7 grade seven, G8 grade eight, n/a not applicable because the education system did not 
participate in this cycle, n/c not comparable to 2015 data 


3.2 Construction of a Proxy Measure for Socioeconomic 
Status 


To address the research questions, we first needed to construct a comparable proxy 
measure for socioeconomic status across the different TIMSS administration cycles. 
The TIMSS home educational resources (HER) index measures important aspects of 
SES, but it is not applicable for trend comparisons across all cycles for several 
reasons. 

First and foremost, the HER index was constructed by different measurement 
methods in different cycles. In 1995 and 1999, the HER index was a simple 
combination of several background variables, including the number of books at 
home, number of home possessions, and parents' education, which were combined 
into three levels: high, medium, and low. For example, students at the high level 
were those with more than 100 books in the home, all three educational possessions 
(computer, study desk, and dictionary), and at least one college-educated parent. 
This index made interpretation easy since each category had its own corresponding 
characteristics. However, since 2011, the HER index has been constructed using IRT 


24 3 Methodology: Constructing а Socioeconomic Index for TIMSS Trend Analyses 


Table 3.2 Home possession items by TIMSS cycle 


Item TIMSS cycle 
1995 1999 2003 2007 2011 2015 
Common Computer | Computer | Computer | Сотршег | Computer Computer/ 
items tablet 
Study Study Study Study desk | Study desk | Study desk 
desk desk desk 
Year-specific | Dictionary | Dictionary | Dictionary | Dictionary | n/a n/a 
items Calculator | Calculator | Calculator | Calculator | n/a n/a 
n/a n/a n/a Internet Internet Internet 
connection | connection | connection 
n/a n/a n/a n/a Own room | Own room 
n/a n/a n/a n/a Books of n/a 
your own 
n/a n/a n/a n/a n/a Own 
mobile 
phone 
n/a n/a n/a n/a n/a Gaming 
system 


Notes n/a item was not present in this cycle of assessment 


scaling methodology (Martin et al. 2011), which allows for the analysis of more 
fine-grained differences in home educational resources between students, and 
enables forward comparability for future administrations even if the components 
of the index should change in the future. The current form of the HER is, however, 
not comparable to the earlier index. In addition to that, in 2003 and 2007, no HER 
index was constructed for TIMSS. 

Second, the components of the HER index changed because the available home 
possession items that students are asked about in the student questionnaire have 
changed over time (Table 3.2). For example, an internet connection at home was not 
part of the questionnaire before 2007, but is now an important component of the 
current HER scale. The only common items across all cycles are a computer and 
study desk. However, in 2015, the question regarding having a computer at home 
was also changed, resulting in two variables: one asking if a student owns a 
computer or tablet at home and a second one asking if a student shares a computer 
or tablet with others in the home. 

It was clear that constructing a consistent measure of SES that can be applied 
across all TIMSS cycles would be of immense value to researchers who wished to 
use TIMSS for trend analyses. We therefore developed a modified version of the 
HER index to address this issue. Our SES measure, which we here term SES*, does 
not represent the full SES construct as usually defined by parental education, family 
income, and parental occupation.* While the construction of such an index serves a 
specific purpose in this study, we believe that the SES* index that is proposed here is 


“An asterisk is added to denote the conceptual difference (Please refer to Chap. 1 for more details). 
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sufficiently closely related to the later IRT-scaled HER versions to yield highly 
relevant and valid results. This index can thus also be beneficially applied to other 
future studies that intend to use the SES* variable for analysis over multiple 
administrations.” 


3.2.1 Components of the SES* Measure 


Our SES* measure, which as mentioned in the introduction is a modified version of 
the HER index, is constructed using three common components across the six cycles 
of TIMSS. These components include (1) number of books at home, (2) number of 
home possessions, and (3) the highest level of education of either parent. 


Number of Books at Home The information is derived from the student questionnaire 
asking how many books students have at home. There are five categories, coded (0) to 
(4): (0) 0 to 10 books; (1) 11 to 25 books; (2) 26 to 100 books; (3) 101 to 200 books; 
and (4) more than 200 books. 


Number of Home Possessions This information comes from questions asking 
students whether they have each of a list of items at home. Since there are only 
two common items (computer and study desk) across all cycles, the total number of 
home possessions ranges from 0 to 2. One caveat needs to be mentioned for 2015. 
The question regarding having a computer at home was changed to two variables: 
one asking if a student owns a computer or tablet at home and the other one asking if 
a student shares a computer or tablet with others at home. We coded a positive 
response to either of these questions as a “1”. Despite the addition of tablet in 2015, 
the correlations of the other SES* components with the computer/tablet variable 
were comparable with those found in 2011 (computer alone), with the scoring of 
either response as a “1”. We therefore believe that the addition of tablet in 2015 did 
not substantially change the construct being measured, and that the SES* index 
remains consistent over time. 


Highest Level of Education of Either Parent 'This is a derived variable constructed 
from both the father's and mother's highest educational levels. The categories of the 
source variables were grouped into five levels in line with the 1995 survey, coded as 
follows: (0) less than lower secondary; (1) completed lower secondary; (2) completed 
upper secondary; (3) postsecondary nontertiary education; and (4) completed 
university or higher. “I don’t know" responses were treated as missing. 


“Ideally, a scaled HER index could be constructed for prior years so that analysis with this index 
would be possible across all TIMSS administrations. However, this exceeds the scope of this 
research project. 
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3.2.2 Multiple Imputation of Missing Values 


The main components of the SES* index have different degrees of missingness. Of 
specific concern is parental education, which on average has missing values of 
around 2096, depending on administration year and education system. Since 
dropping such a large part of the sample would undermine the generalizability of 
the findings, especially when the students with missing values tended to come from 
lower ability levels, multiple imputation was used for all missing values of the SES* 
index components. Instead of imputing the "highest level of parental education" 
variable directly, we imputed father's and mother's education separately, compared 
them after imputation, and then generated the highest level of parental education for 
the SES* index. We imputed the missing values of SES* index variables five times 
using multiple imputation chained equations before constructing the SES* index. 
Imputation using chained equations is known for its flexibility in handling different 
types of variables (for example binary, categorical, and continuous; Hughes et al. 
2014), with our variables of interests being mostly categorical. The imputation is 
achieved by using the observed values for a given individual and the observed 
relations in the data for other participants (Schafer and Graham 2002). 

In addition, since TIMSS data include multiple education systems across multiple 
years, we decided to impute the missing data for each year first and only then create a 
database of all years. The advantage of this approach was that we maximally used 
available information for a given year since the questionnaires have been modified 
over time and thus available relevant variables differ by year. In the imputation 
model, we included all analytic variables that were included in our final analysis, 
other common home possession items available for all education systems in each 
year, plausible values of achievement score, and other related variables (such as 
language spoken at home). After imputation, the correlation between these variables 
in each year was compared between the original dataset and the imputed dataset, and 
the results suggested the imputation preserved the overall relationship among 
variables very well. The student sampling weight was taken into account in the 
imputation model, as shown in a case study of conducting multiple imputation for 
missing data in TIMSS (Bouhlila and Sellaouti 2013). 


3.2.8 The SES* Index 


After imputation, we constructed the SES* index, ranging from 0 to 10 points, by 
assigning numerical values to each category of each of the three components 
(Table 3.3). We applied this to the 13 education systems’ data for the 2011 and 
2015 cycles, and found that this index has a relatively high correlation with the HER 
scale (2011: r — 0.87; 2015: r — 0.84). We also compared the variance in 
mathematics performance explained by the SES* index and by the HER scale in 
2011 and 2015 for these 13 education systems. In 2011, the SES* index explained 
23.796 of the variance in mathematics, while the HER index explained 23.696 of the 
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Table 3.3 SES* index construction 


SES* component Categories Score 
Highest level of parental education Less than lower secondary education 0 
Completed lower secondary education 1 
Completed upper secondary education 2 
Post-secondary, non-tertiary education 3 
Completed university or higher 4 
Home possessions None 0 
Computer/tablet 1 home possession 1 
Study desk 2 home possessions 2 
Number of books at home 0-10 books 0 
11-25 books 1 
26-100 books 2 
101-200 books 3 
More than 200 books 4 


variance. In 2015, the SES* index explained 17.8% of the variance in mathematics, 
while the HER index explained 19.1% of the variance. This suggests that the 
proposed SES* index is highly correlated with the current HER scale and explains 
a similar amount of the variance in students’ achievement. 

The overall weighted distribution and corresponding average mathematics score 
for all participating education systems in 1995 and 2015 suggests that the 
distribution of this index is somewhat left skewed (Figs. 3.1 and 3.2). One 
possible explanation might be that many education systems in our analytic sample 
have an overall high level of SES*. More importantly, the results clearly suggest that 
each additional point in the SES* index is associated with higher average 
mathematics scores. In 1995, the TIMSS achievement score was scaled to have an 
international average value of 500 and a standard deviation of 100 points for 
participating countries. On average, the difference in mathematics scores between 
students with the lowest SES* (0 points) and the highest SES* (10 points) is around 
150 points, which is 1.5 times the standard deviation of TIMSS scores. Furthermore, 
the positive correlation between the SES* index and mathematics scores is not only 
true overall but also holds across all education systems individually. 


3.2.4 Defining High- and Low-SES* Groups 


To calculate the achievement gap between students with high- and low-SES* 
backgrounds over time, we first needed to define the criterion or cut-off points 
corresponding to high- and low-SES* backgrounds. Among the different 
approaches for establishing cut-offs, the main choices are either (a) using common 
cut-offs across educational systems and years, or (b) defining education system- 
specific low-SES* versus high-SES* groups based on the distribution of the SES* 
index for a given year. 


28 3 Methodology: Constructing а Socioeconomic Index for TIMSS Trend Analyses 


20 600 
18 

550 
16 


500 


450 


400 


Weighted 96 of students 
m 
о 
2055 эцешецзеш э8елэлу 


350 


0 1 2 3 4 5 6 7 8 9 10 
SES* index 


Weighted percentage of students Average mathematics score 


Fig. 3.1 Weighted percentage of students and average mathematics score by SES* index, 1995. 
(Note In 1995, 50,332 students in the 13 selected education systems were included in the analysis) 
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Fig. 3.2 Weighted percentage of students and average mathematics score by SES* index, 2015. 
(Note In 2015, 76,159 students in the 13 selected education systems were included in the analysis) 


Common Cut-Offs Given the weighted distribution of the sum-score SES* index for 
all students in all participating education systems across all 20 years, we found that 
an index value of three corresponded to about the 21st percentile of all students, 
whereas a value of eight points corresponded to the 81st percentile (see Table 3.4). 
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Table 3.4 Common cut-offs by overall distribution of SES* index (cumulative proportion) 


SES* Average Australia (1995) Islamic Republic of Iran (1995) 
index percentage cut-off | percentage cut-off percentage cut-off 
0 3 3 22 

1 7 4 44 

2 13 6 63 

3 21 10 76 

4 31 18 84 

5 42 30 91 

6 56 46 95 

7 68 60 97 

8 81 74 99 

9 91 87 100 

10 100 100 100 


As a first test, we applied these cut-off points to all students. Students with three or 
fewer points were defined as the low-SES* group and those with eight or more 
points were defined as the high-SES* group. As can be expected, this approach led to 
very unbalanced groups when the results were examined by education system. For 
example, in Australia in 1995, only 10% of students would have been placed into the 
low-SES* group, while 26% would have been in the high-SES* group. By contrast, 
in Iran, about 76% of students would have been placed in the low-SES* group, with 
only 1% in the high-SES* group (Table 3.4). 


Thus, common cut-offs tend to generate unbalanced groups in certain education 
systems since individual education systems’ specific situations are not taken into 
account. While these may be the actual percentages for high- and low-SES* students 
across educational systems, SES* is a relative concept when viewed within an 
educational system. That is, what is perceived as high or low SES* is society 
dependent. And it is the perception which is important, because what is perceived 
to be real is real in its consequences. Therefore, we decided to establish education 
system specific cut-offs for each year. Given each education system’s distribution of 
SES* in each year, we used quartiles as cut-offs; students in the bottom quartile were 
considered low SES*, while students in the top quartile were considered high SES* 
(see the Appendix for a sensitivity analysis using quintiles versus quartiles and 
additional information). This approach generated better grouping results because it 
takes local context into consideration. 

Another challenge was how to establish exact 25th or 75th percentile cut-offs 
using an index with a range of only 11 points in total. Considering the cumulative 
proportions of students at each SES* point in Australia in 1995 (Table 3.5), we 
found that students with eight points on the index corresponded to the 73th 
percentile, while students with nine points corresponded to the 86th percentile. 
Establishing the bottom quartile was also difficult, since four points corresponded 
to the 15th percentile, while five points corresponded to the 27th percentile. 

To address this issue, we decided to randomly split the sample of students at the 
cut-off point below or above the 25th and 75th percentiles and then combine it with a 
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Table 3.5 Weighted SES* Proportion (96) Cumulative proportion (96) 
distribution of SES* index for o o сре 
Australia, 1995 

1 1 1 

2 2 3 

3 4 6 

4 8 15 

5 13 27 

6 16 43 

7 15 59 

8 14 73 

9 13 86 

10 14 100 


Note The results are based on one of the imputed datasets 


random subsample from the adjacent group, resulting in top and bottom categories 
containing 25% of students. Again, using Australia in 1995 as our example, to obtain 
the bottom quartile, we needed another 10% of students in addition to those having 
0 to 4 points on the SES* index. Therefore, we randomly selected a subsample of the 
Australian students who participated in 1995 and who scored five SES* index points 
to create a sample comprising 25% of students as the bottom SES* category (another 
way to consider this is that if 27% of students are at index point five, then the sample 
contains 2 % more students than needed for the bottom quartile, so 2% of students, in 
absolute terms, have to be randomly excluded from the five-point subsample). 
Applying the same strategy to every individual education system and year 
guaranteed that the bottom- and top-quartile SES* groups always represented 
exactly 25% of students from a given education system in any given year. 


3.3 Analytic Approach 


3.3.1 Plausible Values and Imputed Datasets 


One significant analytic challenge underlying this work was how to simultaneously 
use the existing five plausible values of achievement scores while incorporating 
results from the multiple imputation procedure for the missing values of SES* 
background variables. One approach might be to conduct nested multiple 
imputation, in which the plausible values imputation is nested within the background 
variable imputation (Weirich et al. 2014). However, that would have required an extra 
step back to item responses, and the imputation model would highly depend on the 
final analytic model, meaning that other studies using this SES* index would have to 
create their own models. More importantly, the TIMSS & PIRLS International Study 
Center had clearly stated that principal components for a large number of student 
background variables were included as conditioning variables to improve the reliability 
of the estimated student proficiency scores (Foy and Yin 2016). It is reasonable to 
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believe that the components in our SES* index, which are very important student 
background variables, were included in the TIMSS conditioning models for 
proficiency estimation. Therefore, we used the existing plausible values of 
achievement scores in TIMSS to impute missing values in the SES* component 
variables, together with other relevant variables, resulting in five imputed datasets. 

After imputation, one possibility for using the imputed SES* variable was to 
average the SES* values among the five imputed datasets and thus generate a single 
SES* index score for each student. To validate this approach, we randomly selected 
10% of cases in each country, replaced the existing value of parental education with 
“missing”, imputed the pseudo-missing values using the same imputation model, 
and then compared the imputed values with actual values. However, the validation 
results were not satisfactory, since a simple average of the five imputed values 
presented a quite different distribution from the actual values because it 
overlooked the variance between the imputed values. Therefore, we decided not to 
average the imputed values for SES* but to treat the five imputed SES* values as 
plausible values (Kevin Macdonald, personal communication, 10 March 2018) and 
conduct analyses with the PV module in Stata 14 software (Macdonald 2008). This 
approach allowed us to simultaneously use the five plausible values of the TIMSS 
achievement scores and the five imputed values for the SES* index in the analyses 
for this report. 


3.3.2 Measuring Educational Inequality 


Ferreira and Gignoux (2011) described methods for measuring both inequality in 
achievement (which they saw as being expressed simply by the degree of variability 
in the outcome measure), and inequality in opportunity (they proposed a meaningful 
summary statistic for this would be the amount of variance explained obtained from 
an OLS regression of students’ test scores on a vector C of individual circumstances). 
Another approach was used by Green et al. (2015) in an application using 
international adult skills surveys. Their measure was a “social origins gradient” 
representing the point difference in scores that can be predicted for an individual 
when the education level of his or her parent(s) is increased from the bottom unit to 
the top unit (for example from “less than high school” to “college education"). 

We opted for yet another different approach, one that we believe is better suited 
for trend analysis of educational inequality. To answer the first research question, 
“How has the inequality of education outcomes due to family socioeconomic status 
changed for different education systems between 1995 and 2015”, we calculated the 
achievement gap over time between students in low- and high-SES* groups in terms 
of the average TIMSS achievement score. The larger the gap, the larger the role of 
SES* in determining educational outcomes. 

In addition, we examined whether the changes in achievement gap between high- 
and low-SES* students across years were statistically significant. Since these 
calculations are computationally quite demanding, we focused on providing 
significance testing for changes in achievement gaps only between the following 
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years: (1) 1995 versus 2003, (2) 2003 versus 2015, and (3) 1995 versus 2015. For 
example, to investigate if the change in the gap between 1995 and 2003 was 
statistically significant, the following regression model was conducted: 


f, = Po + В: (SES*) + f; (Year;) + fs (SES* * Year;) + є; 


Where Y, is the predicted achievement score (that is, either mathematics or science) 
for student i in a given education system after controlling for other predictors; ро is 
the mean achievement score for low-SES* students in a given education system in 
1995; f, is the mean score difference between low- and high-SES* students in a 
given education system in 1995; and f); is the coefficient for a categorical variable 
indicating the year of assessment. The reference group is 1995, therefore, the 
coefficient is the mean score difference between students who participated in 2003 
and those in 1995, after controlling for other predictors. Meanwhile J} is the 
coefficient for an interaction term between SES* and the assessment year. This 
reflects how much the achievement gap between low- and high-SES* students in 
2003 differs from the achievement gap in 1995, and, therefore, the p-value for дз 
indicates whether the achievement gap in 2003 is statistically different from the 
achievement gap in 1995. Following the same logic, we conducted similar 
comparisons of the achievement gaps between 2003 and 2015, and between 1995 
and 2015. 

While seeing trends in the SES achievement gaps is important, they can hide 
important changes over time. For example, there might be no change in the size of 
the SES* gap over time because neither group has changed over time, and, in another 
case, the SES* gap may not change because both the lower and upper groups have 
changed in the same direction over time. Because gaps can close or widen for 
different reasons, it is also important to examine how the most disadvantaged 
students are doing over time, as proposed by our second research question, "To 
what extent have education systems managed to increase the academic performance 
of disadvantaged students between 1995 and 2015?" То address this, we analyzed 
the trend in performance among low-SES* students in each education system from 
1995 to 2015. Specifically, we tracked the percentage of low-SES* students who 
performed at or above the TIMSS international intermediate benchmark (that is, 
475 points) for each education system over time. 


3.3.3 Country-Level Indicators in the Educational Systems 
and the Macroeconomic Context 


To better understand our findings in the larger context in which education systems 
operate, we obtained macroeconomic and other indicators from the TIMSS 
encyclopedias as well as data from external sources from 1995 to 2015. The 
external sources we consulted include the World Bank, the UNESCO Institute for 
Statistics, the CIA's World Factbook, the OECD Income Distribution Database, the 
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Indicator 


GDP per person 
(current US$) 


Total percent of 


Source 
The World Bank Open Data 


Link 
https://data.worldbank.org/indica 
tor/NY.GDP.PCAP.CD 


The World Factbook 2018. 
Washington, DC: Central 
Intelligence Agency 


The World Bank Open Data 


https://www.cia.gov/library/publica 
tions/the-world-factbook/index.html 


https://data.worldbank.org/indica 


government tor/se.xpd.totl.gb.zs 

expenditure on The UNESCO Institute for http://data.uis.unesco.org/Index. 
education Statistics aspx ?queryid= 183 

Total percent of GDP | The World Bank Open Data https://data.worldbank.org/indica 


spent on education 


tor/se.xpd.totl.gd.zs 


The UNESCO Institute for 
Statistics 


http://data.uis.unesco.org/? 
queryid—181 


Gini index 


The World Bank Open Data 


https://data.worldbank.org/indica 
tor/SI.POV.GINI 


The World Factbook 2018. 


https://www.cia.gov/library/publica 


Washington, DC: Central 
Intelligence Agency 


The OECD Income Distribution 
Database 


The World Inequality Database 


tions/the-world-factbook/rankorder/ 
2172rank.html 
https://stats.oecd.org/index.aspx? 
queryid—66670 
http://wid.world/data/ 


Top 10% share pre-tax 
national income 


World Inequality Database, and local education agencies (see Table 3.6). We used 
this to interpret our findings against changes in the social context of each education 
system over the 20 years of TIMSS. 


References 


Bouhlila, D. S., & Sellaouti, F. (2013). Multiple imputation using chained equations for missing 
data in TIMSS: A case study. Large-scale Assessments in Education, I(1), 4. 

Ferreira, F. H. G. & Gignoux, J. (2011). The measurement of educational inequality: Achievement 
and opportunity. Policy Research working paper series no. 5873. Washington, DC: The World 
Bank. 

Foy, P., & Yin, L. (2016). Scaling the TIMSS 2015 achievement data. In M. O. Martin, I. V. 
S. Mullis, & M. Hooper (Eds.), Methods and procedures in TIMSS 2015 (pp. 13.1—13.62). 
Chestnut Hill: TIMSS & PIRLS International Study Center, Boston College. Retrieved from 
http://timss.bc.edu/publications/timss/2015-methods/chapter-13.html 

Foy, P., Rust, K., & Schleicher, A. (1996). Sample design. In M. O. Martin & D. L. Kelly (Eds.), 
Third International Mathematics and Science Study (TIMSS) technical report, Volume I: Design 
and development (pp. 87-91). Chestnut Hill: TIMSS & PIRLS International Study Center, 
Boston College. 

Gonzalez, E. J., & Miles, J. A. (Eds.). (2001). TIMSS 1999: User guide for the international 
database. Chestnut Hill: TIMSS & PIRLS International Study Center, Boston College. 
Retrieved from https://timss.bc.edu/timss 1999i/data/bm2_userguide.pdf 


34 3 Methodology: Constructing а Socioeconomic Index for TIMSS Trend Analyses 


Green, A., Green, F., & Pensiero, N. (2015). Cross-country variation in adult skills inequality: Why 
are skill levels and opportunities so unequal in Anglophone countries? Comparative Education 
Review, 59(4), 595—618. 

Hughes, R. A., White, I. R., Seaman, S. R., Carpenter, J. R., Tilling, K., & Sterne, J. A. (2014). Joint 
modelling rationale for chained equations. BMC Medical Research Methodology, 14(1), 1-17. 

LaRoche, S., Joncas, M., & Foy, P. (2016). Sample design in TIMSS 2015. In M. O. Martin, I. V. 
S. Mullis, & M. Hooper (Eds.), Methods and procedures in TIMSS 2015 (pp. 3.1-3.37). 
Chestnut Hill: TIMSS & PIRLS International Study Center, Boston College. Retrieved from 
http://timss.bc.edu/publications/timss/2015-methods/chapter-3.html 

Macdonald, K. (2008). PV: Stata module to perform estimation with plausible values. Statistical 
Software Components 5456951. Boston College Department of Economics, revised 12 Feb 
2014. Retrieved from https://ideas.repec.org/c/boc/bocode/s45695 1 .html 

Martin, M. O., Mullis, I. V., Foy, P., & Arora, A. (2011). Creating and interpreting the TIMSS and 
PIRLS 2011 context questionnaire scales. In M. O. Martin & I. V. S. Mullis (Eds.), Methods and 
procedures in TIMSS and PIRLS (pp. 1-11). Chestnut Hill: TIMSS & PIRLS International 
Study Center, Boston College. Retrieved from https://timssandpirls.bc.edu/methods/pdf/TP 1 1_ 
Context Q Scales.pdf 

Mullis, I. V. S., Martin, M. O., Gonzalez, E. J., Gregory, K. D., Garden, R. A., O'Connor, K. M., 
Chrostowski, S. J., & Smith, T. A. (2000) TIMSS 1999 international mathematics report. 
Findings from IEA's repeat of the Third International Mathematics and Science Study at the 
eighth grade. Chestnut Hill: TIMSS & PIRLS International Study Center, Boston College. 
Retrieved from https://timss.bc.edu/timss1999i/pdf/T99i Math All.pdf 

Mullis, I. V. S., Martin, M. O., Foy, P., & Hooper, M. (2016). TIMSS 2015 international results in 
mathematics. Chestnut Hill: TIMSS & PIRLS International Study Center, Boston College. 
Retrieved from http://timssandpirls.bc.edu/timss2015/international-results 

Schafer, J. L., & Graham, J. W. (2002). Missing data: Our view of the state of the art. Psychological 
Methods, 7(2), 147. 

Weirich, S., Haag, N., Hecht, M., Bóhme, K., Siegle, T., & Lüdtke, O. (2014). Nested multiple 
imputation in large-scale assessments. Large-scale Assessments in Education, 2(1), 9. 


Open Access This chapter is licensed under the terms of the Creative Commons Attribution- 
NonCommercial 4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/), which 
permits any noncommercial use, sharing, adaptation, distribution and reproduction in any medium 
or format, as long as you give appropriate credit to the original author(s) and the source, provide a 
link to the Creative Commons license and indicate if changes were made. 

The images or other third party material in this chapter are included in the chapter's Creative 
Commons license, unless indicated otherwise in a credit line to the material. If material is not 
included in the chapter's Creative Commons license and your intended use is not permitted by 
statutory regulation or exceeds the permitted use, you will need to obtain permission directly from 
the copyright holder. 


Chapter 4 
Socioeconomic Achievement Gaps: Trend Со 
Results for Education Systems 


Abstract This study compared the SES achievement gaps of 13 education systems 
in both mathematics and science over time. Each education system was initially 
categorized as either a system with "increasing SES achievement gaps" or 
"decreasing SES achievement gaps," based on 20-year trends in the IEA's Trends 
in International Mathematics and Science Study (TIMSS). For example, 
achievement gaps between low- and high-SES students significantly widened over 
this time in Hungary, Islamic Republic of Iran, Lithuania, and Singapore, while in 
Norway, Slovenia, and the United States, student achievement gaps in either 
mathematics or science narrowed between 1995 and 2015. Education system 
specific graphs show trends in the achievement gaps between high- and low-SES 
students, as well as trends for low-SES students reaching the TIMSS intermediate 
benchmark over time. Monitoring intermediate changes between 1995 and 2003, 
and between 2003 and 2015, as well as in-depth analysis of relevant country-specific 
macroeconomic indicators for each education system, provides further context. 


Keywords Achievement gaps - Disadvantaged students - Educational inequality - 
International large-scale assessment - Macroeconomic indicators - Socioeconomic 
status (SES) - Trends in International Mathematics and Science Study (TIMSS) 


4.1 Overall Findings 


Our first research question asked to what extent inequality of education outcomes 
due to 5Е5“! had changed for the 13 education systems from 1995 to 2015. To 
answer this, we calculated the achievement gap over time between students 


'The SES measure used in this study is a modified version of the TIMSS home educational 
resources (HER) index and does not represent the full SES construct, as usually defined by parental 
education, family income, and parental occupation. In this report, we therefore term our measure 
SES* to denote the conceptual difference (Please refer to Chap. 1 for more details). 


O International Association for the Evaluation of Educational Achievement (IEA) 2019 35 
M. Broer et al., Socioeconomic Inequality and Educational Outcomes, IEA Research for 
Education 5, https://doi.org/10.1007/978-3-030-11991-1 4 
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categorized in high- and low-SES* groups in terms of their average mathematics and 
science scores. Larger gaps suggest that SES* plays a larger role in educational 
outcomes in that education system. 

Some systems presented a linear change, while others experienced some ups and 
downs in terms of trend. Among the 13 education systems selected for study, we 
observed two general patterns: overall, SES* achievement gaps were either 
increasing or decreasing, although there were variations in the magnitude of 
differences (Table 4.1). 

As mentioned, to provide better context for understanding the results, we also 
looked in-depth at the country-specific findings. The educational and 
macroeconomic context is critical to an understanding of trend changes in SES* 
gaps, as well as the changes between intermediate cycles of TIMSS (readers should 
refer to Table 3.6 for further information on the sources for such data). We also 
determined what percentage of low-SES* students achieved at or above the TIMSS 
international intermediate benchmark for mathematics and science (475 points; 
Tables 4.2 and 4.3). 


4.1.1 Increasing SES* Achievement Gap 


Six education systems demonstrated an overall increase in the achievement gap in 
mathematics between high- and low-SES* students between 1995 and 2015 
(Fig. 4.1); for three education systems (Hungary, Iran, and Singapore), the results 
are Statistically significant. For example, in Hungary, the achievement gap in 
mathematics was already high at 103 points in 1995, and had increased to 
143 points by 2015. This was an improvement of about one-half of a standard 
deviation (100 points). 

Similarly, our results show the achievement gap in science between high- and 
low-SES* students increased in Hungary, Iran, and Lithuania, confirming this trend 
(Fig. 4.2). For instance, in Iran, the difference in the science score between high- and 
low-SES* students increased from 41 points in 1995 to 104 points in 2015. This 
suggests that family background plays an increasingly important role regarding 
students’ academic outcomes in Iran. 


4.1.2 Decreasing SES* Achievement Gap 


Seven education systems seem to have experienced an overall decrease in the 
achievement gap in mathematics between high- and low-SES* students between 
1995 and 2015 (Fig. 4.3). However, we only identified statistically significant 
changes in Norway and Slovenia. As an example, the mathematics achievement 
gap between high- and low-SES* students in Norway was 63 points in 1995, 
declining to 50 points in 2015. 
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Fig. 4.1 Education systems showing an increasing mathematics score difference between high- 
and low-SES* students over 20 years of TIMSS 
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Fig. 4.2 Education systems showing an increasing science score difference between high- and 
low-SES* students over 20 years of TIMSS 


Correspondingly, we found that among the six education systems that recorded an 
overall decrease in the SES* gap for science achievement (Fig. 4.4), the decrease 
was only significant for the United States. The mean score difference in science 
between high- and low-SES* students in the United States was 105 points in 1995 
and 82 points in 2015, indicating that the achievement gap in science has been 
narrowing over time. 
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Fig. 4.3 Education systems showing a decreasing mathematics score difference between high- and 
low-SES* students over 20 years of TIMSS 
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Fig.4.4 Education systems showing a decreasing science score difference between high- and low- 
SES* students over 20 years of TIMSS 


4.0 Education System Specific Findings 
4.2.1 Australia 


Australia has a decentralized education system, and responsibilities are held 
primarily by the states and territories. Australia offers a 10-year compulsory 
education for all children. The system is structured as seven or eight years of 
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primary school, including kindergarten through to grade six or seven. Lower 
secondary school runs for three or four years, and upper secondary education 
comprises two years, from grade 11—12. Public schools are fully funded by the 
government, while private schools are partially funded by the government and also 
charge fees (Australian Government 2018). 

Australian students have participated in every cycle of TIMSS since its inception 
in 1995. The average mathematics and science performance of Australian students 
has remained more or less the same over time. We categorized Australia as one of the 
countries that witnessed an overall decrease in the achievement gap between high- 
and low-SES* students. However, a more precise description could be that the gap 
has not widened since 1995. For both mathematics and science, there was a 
substantial decrease in the gap between 1995 and 2003, by thereafter there was a 
considerable increase (Figs. 4.5 and 4.6). If we consider science scores, the 
achievement gap between high- and low-SES* students was 97 points in 1995 and 
69 points in 2003. However, the gap increased to 91 points in 2007 and 92 points in 
2011, before dropping again to 84 points in 2015. Overall, despite the clear increase 
since 2003, the gap in 2015 remains lower than it was in 1995. Note that our 
significance test results (Table 4.1) also show that the decrease in the SES* 
science achievement gap between 1995 and 2003 (28 points) and the increase 
between 2003 and 2015 (15 points) are statistically significant, while the 
difference between 1995 and 2015 (13 points) is not. 
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Fig. 4.5 Percentage of low-SES* students achieving at or above the TIMSS intermediate 
benchmark and the average mathematics score for low- and high-SES* groups for Australia 
(1995-2015) 
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Fig. 4.6 Percentage of low-SES* students achieving at or above the TIMSS intermediate 
benchmark and the average science score for low- and high-SES* groups for Australia (1995—2015) 


This pattern was also reflected by the percentage of low-SES* students achieving 
at or above the intermediate benchmarks in both mathematics and science in 
Australia. For example, in science, 50% of low-SES* students achieved at the 
intermediate benchmark in 1995 and the percentage increased to 5996 in 2003. 
However, after 2003, there was a noticeable decrease, and only about 50% of 
low-SES* students achieved at the intermediate benchmark across the last three 
cycles. In other words, Australia observed a significant decrease in educational 
inequality during the first decade of TIMSS, but this trend apparently reversed in 
the second decade. 

More broadly, the gross domestic product (GDP) per person in Australia has been 
steadily increasing over the last two decades, from around US$ 20,400 in 1995 to 
US$ 56,400 in 2015. The percentage of Australian GDP spent on education has 
remained around 5%. Furthermore, the Gini index (which assesses the level of 
inequality in the wealth of a nation; Gini 1921) for Australia remained at the 
33-points level from 1995 to 2003, suggesting that there was no significant change 
in income inequality over that period. However, the Gini index increased to 35 in 
2008 and remained at that higher level in 2010, suggesting a later increase in income 
inequality. This was also reflected by a stable increase in the top 10% of earners’ 
share of pre-tax national income, which went from 2746 in 1995, to 2996 in 2003, to 
32% by 2015. In summary, the country-level economic indicators seem to be in 
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accordance with our findings, suggesting that Australia observed a strong, significant 
decrease in educational inequality during the first decade of TIMSS, but that this did 
not continue into the second decade. 


4.2.2 Hong Kong 


Hong Kong is one of the richest education systems in the world with a high GDP per 
person (OECD 2011), over US$ 42,000 in 2015. During the period 1995-2015, the 
education system experienced many changes, especially after the transfer of its 
sovereignty from the United Kingdom to China in 1997. Until the 2009—2010 
academic year, Hong Kong's education system was very similar to the British 
education systems. It subsequently changed to the Chinese system of six years of 
primary school, three years of junior (lower) secondary school, three years of senior 
(upper) secondary school, and four years of college. The government provides a free 
and universal education for the first O years of schooling to all children attending 
public schools. From the 2008-2009 academic year onwards, senior secondary 
education has also been free for all public school students (Hong Kong Education 
Bureau 2018). 

Hong Kong have participated in every cycle of TIMSS since 1995 and their 
students have consistently achieved scores comparable with the top education 
systems in the TIMSS assessment. Our analyses show that students in Hong Kong 
performed remarkably well on the TIMSS mathematics and science assessments 
regardless of their SES* background. The average scores for low-SES* students for 
both mathematics and science are higher than the international intermediate 
benchmark in any given year (Figs. 4.7 and 4.8). 

For both mathematics and science, the achievement gaps between high- and 
low-SES* students narrowed prior to 2003 but subsequently increased strongly. 
For example, the achievement gap in mathematics decreased from 53 points in 1995 
to 35 points in 1999, and 41 points in 2003; however, this was not a statistically 
significant decline (Table 4.1). The achievement gap widened to 64 points in 2007, 
74 points in 2011, and 63 points in 2015. The 22-point increase from 2003 to 2015 is 
statistically significant (Table 4.1). As in Australia, there was a considerable 
decrease followed by a substantial increase. However, in contrast to Australia 
(where the later gaps never surpassed the gap in 1995), Hong Kong's subsequent 
increase in the SES* achievement gap in the second decade of TIMSS was much 
larger. 

Moreover, for both mathematics and science, the percentage of low-SES* 
students achieving at or above the TIMSS intermediate benchmark indicate the 
same trends. As mentioned, low-SES* students in Hong Kong have performed 
remarkably well in international comparisons. This is reflected by the consistently 
high percentage of low-SES* students achieving at or above the intermediate 
benchmark in mathematics. In 1999 and 2003, 89% of low-SES* students 
surpassed the benchmark, while only 7996 did so in 2011; however, these 
percentages still surpass those of most other education systems. Hong Kong 
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Fig. 4.7 Percentage of low-SES* students achieving at or above the TIMSS intermediate 
benchmark and the average mathematics score for low- and high-SES* groups for Hong Kong 
(1995-2015) 


100 700 
2 650 
80 
600 
2 70 Е 
2 d 
(и < 
S 60 550 5 
i б 
чл ô. 
о Ф 
x 50 500 8 
9 40 А 
5 450 8 
Ф 
= 30 
400 
20 
350 
10 
0 300 
1995 1999 2003 2007 2011 2015 
Year 


96 Low SES* students achieving at or above the TIMSS intermediate benchmark 
Average science score for low SES* students 


Average science score for high SES* students 


Fig. 4.8 Percentage of low-SES* students achieving at or above the TIMSS intermediate 
benchmark and the average science score for low- and high-SES* groups for Hong Kong 
(1995-2015) 
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students, both from low- and high-SES* backgrounds, continued to perform very 
well, although there was evidence of a widening SES* achievement gap and growing 
educational inequality in the second decade of TIMSS. 

One possible explanation may be related to macro-level changes in the Hong 
Kong economy. For example, the percentage of total government expenditure 
allocated to education increased from 1796 in 1996 to 2396 in 2003, but dropped 
to 19% in 2015. Similarly, GDP spending on education was 2.796 in 1995, increased 
to 4.396 in 2003, and then dropped to 3.3% in 2015. Meanwhile, the Gini index was 
43 in 1996 and 54 in 2011, suggesting increasing income inequality. In summary, 
the overall trend in income inequality mirrored the trend in educational inequality. 


4.2.3 Hungary 


Hungary has a decentralized education system with shared responsibilities among 
national, local, and institutional levels. Until age 18, students are mandated to 
receive compulsory education. For the years of compulsory education, public 
schools are offered free to all students while private schools charge fees. The 
education system in Hungary is structured as eight years of elementary education, 
followed by four years of secondary education. Secondary education typically 
covers grades 9-12, and provides academic or vocational oriented programs. 

Hungary has participated in every cycle of TIMSS since its inception in 1995. 
The average performance in mathematics and science among Hungarian grade eight 
students has been declining over time (Mullis et al. 2016). However, we found that 
Hungary also experienced a very strong increase in the achievement gap between 
students from high- and low-SES* groups. There was a 31-point increase in the gap 
in science (from 93 in 1995 to 124 in 2015) and a 41-point increase in the gap in 
mathematics (from 103 in 1995 to 144 in 2015) over the 20 years of TIMSS 
(Tables 4.2 and 4.3). From 2003 to 2007, while Hungarian students' overall 
performance declined, note that the decrease was much more extensive for 
low-SES* students than for high-SES* students. For instance, in mathematics, 
there was a 28-point decrease (from 484 to 456) for low-SES* students but only 
an eight-point decrease (from 580 to 572) for high-SES* students (Fig. 4.9). 
Meanwhile, from 2011 to 2015, the average mathematics score for high-SES* 
students increased significantly, while it remained the same for low-SES* 
students. The significance test results show that the minor change between 1995 
and 2003 is not statistically significant while the 48-point increase in the 
achievement gap between 2003 and 2015 is statistically significant (Table 4.1). 

In addition, the percentage of low-SES* students achieving at or above the 
mathematics and science TIMSS intermediate benchmark is in agreement with the 
achievement gap trend (Figs. 4.9 and 4.10). The percentage of low-SES* students 
achieving at or above the mathematics intermediate benchmark decreased 
significantly, from 51% in 1995 to 34% in 2015 (Fig. 4.9). Over the period 
1995-2015, a decline in performance can be observed for all students in Hungary, 


4.2 Education System Specific Findings 47 


but the decline was more substantial for low-SES* students, resulting in a higher 
inequality in educational outcomes. 
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Fig. 4.9 Percentage of low-SES* students achieving at or above the TIMSS intermediate 
benchmark and the average mathematics score for low- and high-SES* groups for Hungary 
(1995-2015) 
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Fig. 4.10 Percentage of low-SES* students achieving at or above the TIMSS intermediate 
benchmark and the average science score for low- and high-SES* groups for Hungary (1995-2015) 
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At the macro-level, Hungary has a relatively low GDP per person compared to 
other participating countries. By 2015, the current GDP per person for Hungary was 
a little over US$ 12,000. The percentage of total government expenditure allocated 
to education was 1096 in 1996, increased to 1296 in 2003, and then dropped back to 
9% in 2011 and 2014, which is considerably lower than in many other countries. 
These trends were also reflected in the percent of GDP spent on education, which 
was 4.990 in 1995, peaked at 5.896 in 2003, and then dropped back to 4.796 in 2014. 
Meanwhile, Hungary experienced a considerable increase in its Gini index from 
28 in 1993 to 35 in 2005, suggesting a growing income inequality in its society. The 
increase in the Gini index is also supported by another indicator, which is the top 
10% share of pre-tax national income. The available data from 1995 to 2008 shows 
that the top 1096 of earners' share of national income increased strongly from 2596 to 
3290. In summary, the evidence of growing income inequality in Hungary is in 
alignment with our findings suggesting that the achievement gap between high- and 
low-SES* students in Hungary has increased significantly over time. 


4.2.4 Islamic Republic of Iran 


Iran has a centralized education system where the Ministry of Education regulates all 
schools and holds financial responsibilities for primary and secondary education. The 
system is structured as five years of primary education, three years of secondary 
education (also called the "guidance cycle"), three years of upper secondary 
education, and one year of pre-university education or two years of associate degree 
education (Nuffic 2015). The 3 years of secondary education in the guidance cycle is 
used to differentiate whether a student best suited for an academic or vocational upper 
secondary education. Iran has participated in every cycle of TIMSS since 1995; 
TIMSS is administered to Iranian students at a critical time point in their education, 
immediately before they are separated into different tracks for their upper secondary 
education. Overall, the performance of Iranian students in mathematics and science 
has been below the international average. Of all the education systems we considered 
in this study, Iran is also the country with the lowest GDP per person. 

The achievement gap, however, has been large and has significantly expanded over 
the 20 years of TIMSS. The mathematics score difference between high- and low-SES* 
students increased from 45 points in 1995, and 61 points in 2003, to 106 points in 2015 
(Table 4.2). The significance test suggests that the 16-point increase between 1995 and 
2003, and the 45-point increase between 2003 and 2015 are both statistically significant 
(Table 4.1). The average mathematics score for high-SES* students has been increasing 
over time, from 443 points in 1995 to 495 points in 2015, whereas the performance for 
low-SES* students has simultaneously been decreasing, from 398 points in 1995 to 
388 points in 2015 (Fig. 4.11). These opposing trends suggest that the low-SES* 
students have been increasingly left behind, with SES* achievement gaps widening 
and educational inequality worsening. A similar trend is observed in science, with 
high-SES* students’ performance remaining the same from 1995 to 2003, increasing 
significantly between 2003 and 2011, and decreasing slightly between 2011 and 2015 
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Fig. 4.11 Percentage of low-SES* students achieving at or above the TIMSS intermediate 
benchmark and the average mathematics score for low- and high-SES* groups for the Islamic 
Republic of Iran (1995-2015) 


(Fig. 4.12). In contrast, low-SES* students’ performance has been on a downward 
trajectory since 1995. 

Moreover, the percentage of low-SES* students achieving at or above the TIMSS 
intermediate benchmark reinforces the same story of declining performance among 
students in the bottom SES* quartile. In 1995, only 1696 of students achieved at or 
above the benchmark in mathematics, with that number dropping to less than 1096 in 
2003, 2007, and 2011 (Fig. 4.11). For science, the percentage of low-SES* students 
achieving at or above the TIMSS intermediate benchmark dropped from 32% in 
1995 to 19% in 2015 (Fig. 4.12). The strong decline in the percentage of low-SES* 
students achieving at or above the TIMSS intermediate benchmark supports our 
observation of a widening SES* achievement gap in Iran. 

At the macro-level, Iran's economy has been negatively influenced by 
international sanctions and regional tensions since the 1980s. The GDP per person 
has increased, from US$ 1592 in 1995 to US$ 7729 in 2011, but this was 
accompanied by a significant decline to US$ 4862 in 2015. The percentage of 
total government expenditure allocated to education increased from 15% in 1995 
to 22% in 2003; this has subsequently decreased, and was 17% at its lowest point in 
2012. In contrast, the percentage of GDP spent on education has continuously 
decreased from 4.4% in 1995 to 3% in 2014. Finally, the Gini index has remained 
high from 1994 to 2016, with one of its lowest points occurring at 39 in 2016. In 
summary, Iran has been spending less on education and retained a high Gini index, 
which suggests large income inequalities. This also supports our findings of an 
increasing achievement gap, with low SES* students steadily falling behind 
since 1995. 
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Fig. 412 Percentage of low-SES* students achieving at or above the TIMSS intermediate 
benchmark and the average science score for low- and high-SES* groups for the Islamic Republic 
of Iran (1995-2015) 


4.2.5 The Republic of Korea 


The Republic of Korea has a decentralized education system in which the local 
authorities play a significant role in planning the budget and making decisions. The 
school system comprises six years of primary school, three years of middle school, 
three years of high school, and four years of university. It offers nine years of free 
and compulsory education and implements a randomized school assignment system 
at the primary- and middle-school level within each school district (Lee 2001). 

The Republic of Korea has participated in every cycle of TIMSS since 1995 and 
has performed among the top education systems in both the mathematics and science 
assessments. Similar to their peers in Hong Kong, students in the bottom SES* 
quartile in the Republic of Korea have also demonstrated a comparatively high 
achievement. The average mathematics score for low-SES* students was close to 
or even exceeded the TIMSS high-performing benchmark (550 points). However, 
the SES* achievement gap is still striking, considering the education system's 
generally high performance. 

In terms of mathematics achievement, the gap between high- and low-SES* 
students remained almost unchanged in the first decade of TIMSS, going from 
94 points in 1995, and 82 points in 1999, to 93 points in 2003. In the second 
decade of TIMSS, the achievement gap increased to 106 points in 2007 and 
107 points in 2011. Although the achievement gap subsequently declined to 
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Fig. 4.13 Percentage of low SES* students achieving at or above the TIMSS intermediate 
benchmark and the average mathematics score for low- and high-SES* groups for the Republic 
of Korea (1995-2015) 


84 points in 2015, this was not due to an improvement in the performance of 
low-SES* students, but rather a decline in the achievement of high-SES* students 
(Fig. 4.13). Similarly, very minor changes were found in the science achievement 
gap (Fig. 4.14); in 2015, again the achievement gap closed slightly as the 
performance of high-SES* students had declined. However, we found that the 
change in the science achievement gap was not statistically significant (Table 4.1). 

Furthermore, the bar graphs confirm that the changes in the mathematics and 
science achievement gaps in 2015 were mainly due to a decrease in high-SES* 
students’ performance since the percentage of low-SES* students achieving at the 
TIMSS intermediate benchmark remained the same. Everything considered, Korean 
students on average have performed very well on the TIMSS assessment, but the 
SES* achievement gap is still sizable. 

At the macro-level, the Republic of Korea experienced long-term stable economic 
development, with GDP per person increasing from US$ 12,333 in 1995 to US$ 
27,105 in 2015, except for the significant decline around the time of the 1998 
economic crisis. The government has also expanded support for education by 
spending more GDP on education over time. In 1995, only 396 of GDP was spent 
on education, but this had increased to 4.9% Бу 2011. However, income inequality 
has widened over time, as reflected by the top 1096 of earners' share of the national 
income, which increased from 29% in 1995 to 44% in 2011. This accords with the 
increasing students’ SES* achievement gap that we identified from 20 years of 
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Fig. 4.14 Percentage of low-SES* students achieving at or above the TIMSS intermediate 
benchmark and the average science score for low- and high-SES* groups for the Republic of 
Korea (1995-2015) 


TIMSS data. One possible link between the widening income gap and the increasing 
SES* achievement gap might be the private shadow education in the Republic of 
Korea, which is common practice for students and requires intensive parental 
involvement and financial support (Byun and Kim 2010). 


4.2.6 Lithuania 


Lithuania has established a decentralized education. system, where school 
governance is distributed among state-, municipal-, and school-level authorities. 
Most schools are public schools, and offer a free and compulsory education for 
children between seven and 16 years of age. The Lithuanian education system 
consists of four years of primary education, six years of lower secondary 
education, and two years of upper secondary education (OECD 2017). 

Lithuania has participated in every cycle of TIMSS from its inception in 1995. 
The average performance of Lithuanian students has increased by 40 points in 
mathematics and 58 points in science between 1995 and 2015, an improvement of 
about one-half of a standard deviation (100 points). Meanwhile, the increase in 
mathematics performance for high- and low-SES* students has remained essentially 
the same, suggesting the achievement gap has been stagnant (Fig. 4.15). 
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Fig. 4.15 Percentage of low-SES* students achieving at or above the TIMSS intermediate 
benchmark and the average mathematics score for low- and high-SES* groups for Lithuania 
(1995-2015) 


The changes in mathematics performance were not statistically significant 
(Table 4.1). In contrast, the SES* achievement gap in science showed a different 
trend, decreasing from 73 points in 1995 to 54 points in 2003, and then increasing to 
90 points in 2015 (Fig. 4.16). This 20-point decrease in the first decade of TIMSS, as 
well as the subsequent 36-point increase were both statistically significant, which 
resulted in an overall widening achievement gap between 1995 and 2015 
(Table 4.1). 

The percentage of low-SES* students achieving at or above the TIMSS 
mathematics and science intermediate benchmark deliver a more positive message. 
Note that only about 30% of students in Lithuania achieved at or above the TIMSS 
intermediate benchmark in mathematics in the first two cycles of the TIMSS 
assessment, while almost 50% performed at or above the benchmark in 2015 
(Fig. 4.15). Similarly, 27% of low SES* students achieved at or above TIMSS 
intermediate benchmark in science in 1995, with that figure almost doubling in 
2015 to 53% (Fig. 4.16). Although the SES* achievement gap increased in the 
second decade of TIMSS, increasing numbers of students at the lowest SES* quartile 
have been reaching the intermediate benchmark over time. 

More broadly, Lithuania has experienced a strong economic improvement since 
1995, with GDP per person increasing from US$ 2169 in 1995 to US$ 14,289 in 
2015. Meanwhile, the government has been spending less on education as a share of 
the total government expenditure over time: in 1995, 21% of the government’s total 
expenditure was spent on education, while this decreased to 16% in 2003, 
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Fig. 4.16 Percentage of low-SES* students achieving at or above the TIMSS intermediate 
benchmark and the average science score for low- and high-SES* groups for Lithuania 
(1995-2015) 


and continued to drop to 13% in 2013. In addition, the Gini index in Lithuania 
increased from 34 in 1993 to 38 in 2014, indicating widening income inequality over 
time. This is in alignment with our finding that there is a noticeable increase in the 
SES* achievement gap for both mathematics and science between 2003 and 2015. 


4.2.7 New Zealand 


New Zealand has a decentralized education system with compulsory education for 
all children usually aged from five or six years to age 16 (grade 11). The four 
separate levels in the education system are early childhood, primary, secondary, and 
tertiary education. In New Zealand, authority for daily processes and financial 
management is decentralized from the government onto its educational institutions 
(Caygill and Flockton 2016). New Zealand has participated in most cycles of TIMSS 
since 1995, with the exception of the 2007 cycle. Over time, students have been 
improving their performance in both mathematics and science. 

In New Zealand, the SES* achievement gap for mathematics declined in the first 
decade of TIMSS, but increased significantly after 2003 (Fig. 4.17). The 
achievement gap between high- and low-SES* students in mathematics was 
82 points in 1995, and had decreased to 70 points by 2003. This 12-point change, 
however, was not statistically significant (Table 4.1). After 2003, the achievement 
gap increased to 78 points in 2011, and eventually reached 96 points in 2015. The 
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Fig. 4.17 Percentage of low-SES* students achieving at or above the TIMSS intermediate 
benchmark and the average mathematics score for low- and high-SES* groups for New Zealand 
(1995-2015) 


26-point difference in the achievement of high- and low-SES* students between 
2003 and 2015 was statistically significant (Table 4.1). 

The SES* achievement gap in science performance experienced a corresponding 
ebb and flow (Fig. 4.18). It was 93 points in 1995 and dropped to 78 points by 2003. 
In the second decade of TIMSS, however, the achievement gap increased to 
85 points in 2011, and eventually reached 106 points in 2015, which was a 
statistically significant change (Table 4.1). 

Furthermore, the percentage of low-SES* students achieving at or above the 
TIMSS intermediate benchmark for mathematics decreased over time, from 45% in 
1995, to 36% in 1999, and 41% in 2003. This then dropped to 35% in 2011, and 34% 
in 2015. This suggests that the increasing achievement gap in the second decade was 
due to fewer low SES* students achieving at or above the TIMSS intermediate 
benchmark, as well as a strong improvement in high SES* students’ performance. 

In general, the GDP per person in New Zealand has been progressively increasing 
from US$ 17,400 in 1995 to US$ 38,202 in 2015. Similarly, annual government 
expenditure on education has been continuously growing, from almost 16% of the 
total government expenditure in 1995 to 18% in 2015. This concurs with the 
percentage of GDP spent on education, which has remained around six to seven 
percent, and is higher than the majority of other countries. The top 10% of earners’ 
share of pre-tax national income in New Zealand has remained at 32% over the past 
two decades, suggesting no significant change in income inequality. Future work 
should focus on other factors that might explain the increasing achievement. 
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Fig. 4.18 Percentage of low-SES* students achieving at or above the TIMSS intermediate 
benchmark and the average science score for low- and high-SES* groups for New Zealand 
(1995-2015) 


4.2.8 Norway 


Norway offers a centralized education curriculum for grades one to 13 in all subjects. 
Access to public education is free for all students. Compulsory education usually 
starts at age six years and continues to age 15 or 16 years old. Most students in 
Norway are enrolled in publicly funded schools. There are three stages in the 
Norwegian education system. First is the “child stage," comprised of grades one to 
seven. Then there is the “youth stage,” comprised of grades eight to 10 (Norwegian 
Directorate for Education and Training 2014), followed by a non-compulsory upper 
secondary school stage (grades 11-13). Sometimes, due to Norway’s dispersed 
population, these stages may be either all delivered by one school or taught in 
schools separated by these stages. 

Norway has participated in most cycles of TIMSS since 1995, with the exception 
of the 1999 cycle. Norway observed an overall decrease in the achievement gap for 
mathematics between high- and low-SES* students over time. More specifically, 
there was no significant change in the gap between 1995 and 2003. After 2003, the 
gap decreased from 63 points to 56 points in 2007, and 50 points in 2015, with an 
intermediate increase in 2011 (Fig. 4.19). Despite this minor volatility in the trend, 
we consider there to be an overall decreasing trend over the time period 1995 to 2015 
because, in 2015, the gap reached its lowest point. Our results showed that the 
observed 13 point-decrease was statistically significant (Table 4.1). 
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Fig. 4.19 Percentage of low-SES* students achieving at or above the TIMSS intermediate 
benchmark and the average mathematics score for low- and high-SES* groups for Norway 
(1995-2015) 


The change in the percentage of low-SES* students achieving at or above the 
TIMSS mathematics intermediate benchmark provides more information about the 
trend. For example, the achievement gap remained the same from 1995 to 2003. 
However, in 2003, fewer low-SES* students achieved at or above the TIMSS 
intermediate benchmark than in 1995, due to an overall decline in mathematics 
performance for all students in Norway. In contrast, the decline in the gap in 2015 
between high- and low-SES* students is due more to improvement in low-SES* 
students’ performance than high-SES* students’ performance. 

Conversely, for science, the SES* achievement gap is not statistically significant 
and remained essentially the same (about 60 points) between 1995 and 2003 
(Fig. 4.20). There was then a considerable increase to 72 points in 2007, and 
80 points in 2011, followed by a sizable decline to 66 points in 2015. The 
difference in the gaps between 1995 and 2015, and 2003 to 2015 were not 
statistically significant (Table 4.1). 

As in mathematics, there was no change in the SES* achievement gap in science 
between 1995 and 2003, due to an overall decline in performance for all students 
in Norway. The increase in the achievement gap in 2007 occurred because of a 
decline in low-SES* students’ performance, with less than 40% of low-SES* 
students achieving at or above the TIMSS intermediate benchmark in 2007, while 
high-SES* students’ performance remained the same. Moreover, between 2007 and 
2011, low-SES* students’ performance in science did not improve as much as that of 
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Fig. 4.20 Percentage of low-SES* students achieving at or above the TIMSS intermediate 
benchmark and the average science score for low- and high-SES* groups for Norway (1995-2015) 


high-SES* students. The percentage of low-SES* students whose achievement was 
at or above the TIMSS intermediate benchmark remained the same. Therefore, the 
observed widening of the SES* achievement gap from 2007 to 2011 was due mainly 
to an improvement in high-SES* students’ performance. After that, the gap declined 
from 2011 to 2015 because of a decline in high-SES* students’ performance and a 
slight increase in low-SES* students’ performance. 

By and large, the GDP per person in Norway has been gradually increasing over 
the last two decades, from US$ 34,875 in 1995 to US$ 74,498 in 2015. The share of 
total government expenditure spent on education remained the same, at 17% of the 
total government expenditure in 1995 and 2013. This is in agreement with the 
percentage of GDP spent on education, which has also remained constant at 
around 7%, a figure that is significantly higher than most of its international 
counterparts. Additionally, in the second decade of TIMSS, the Gini index for 
Norway suggested a significant decline in income inequality, decreasing from 
32 in 2004 to 27 in 2014. This is partially consistent with our finding that there 
was a significant decrease in the SES* achievement gap in mathematics. 


4.2.9 Russian Federation 


The Russian Federation provides 11 years of compulsory education for all children, 
usually from the age of six-and-a-half years up to age 16. Grade one is the first year 
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of compulsory education, while grade 11 is the last year of secondary education and 
compulsory schooling. Primary education comprises grades one to four, lower 
secondary education includes grades five to nine, and upper secondary education 
consists of grades 10 and 11. From 2000, the Russian government commenced its 
development of a new educational reform program, with an aim to provide higher 
quality education, as well as more equal access (Kovaleva and Krasnianskaia 2008). 
Overall, in terms of policymaking and funding, the Russian education system has 
become more decentralized, with federal education authorities responsible for 
developing and overseeing federal/educational standards, policy, and legislation, 
as well as those of federal subjects (republics). 

The Russian Federation has participated in every cycle of the TIMSS assessment. 
The Russian Federation can be categorized as one of the education systems that 
witnessed an overall decrease in the SES* achievement gap, although the difference 
in the gap between 1995 and 2015 is not statistically significant (Table 4.1). For 
mathematics, the SES* achievement gap remained the same between 1995 and 2003, 
and then significantly decreased between 2003 and 2015. The achievement gap 
between high- and low-SES* students in mathematics was 63 points in 1995 and 
72 points in 1999, followed by a small decrease to 66 points in 2003 (Fig. 4.21). The 
gap increased to 81 points in 2007, then decreased again to 67 points in 2011, before 
falling further to 47 points in 2015. 

The trend is also reflected by the percentage of low-SES* students achieving at or 
above the TIMSS intermediate benchmark in the Russian Federation for mathematics 
in the first decade of TIMSS. The percentage of low-SES* students achieving at or 
above the TIMSS intermediate benchmark declined from 1995 to 2003. In 1995, 6096 
of low-SES* students achieved at or above the TIMSS intermediate benchmark for 
mathematics, while this number declined to 57% in 1999, and 49% in 2003. The 
percentages increased to 5096 of low-SES* students achieving at or above the TIMSS 
intermediate benchmark in 2007, 6696 in 2011, and 6996 in 2015, partially explaining 
the decreasing SES* achievement gap in the second decade of TIMSS. 

For science, there appears to have been no noticeable change in the achievement 
gap for the first decade of TIMSS, and a substantial decrease is noted for the last two 
cycles of the TIMSS assessment (Fig. 4.22). There was also a decrease in the 
percentage of low-SES* students achieving at or above the TIMSS intermediate 
benchmark from 1995 to 2003. This suggests that the relatively consistent 
achievement gap from 1995 to 2003 was due to an overall decrease in both high- 
and low-SES* students" performance. In the TIMSS cycles after 2003, an increasing 
percentage of low-SES* students achieved at or above the TIMSS intermediate 
benchmark, explaining the decreasing trend in the SES* achievement gap between 
2003 and 2015. 

GDP per person has been steadily increasing in the country over the last two 
decades from around US$ 2700 in 1995 to US$ 9329 in 2015. The percentage of 
GDP spent by government on education was low, at around three to four percent, 
indicating a relatively low level of investment in education compared with other 
education systems. However, the Gini index went from 46 in 1996 to 37 in 2002, 
suggesting that there was a significant reduction in income inequality during that 
period. This decline then reversed in 2007, when the Gini index increased to 42, but 
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Fig. 4.21 Percentage of low-SES* students achieving at or above the TIMSS intermediate 
benchmark and the average mathematics score for low- and high-SES* groups for the Russian 
Federation (1995-2015) 
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Fig. 4.22 Percentage of low-SES* students achieving at or above the TIMSS intermediate 
benchmark and the average science score for low- and high-SES* groups for the Russian Federation 


(1995-2015) 
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by 2015 it had again dropped to 38 points. In conclusion, country-level indicators 
support our findings suggesting that the Russian Federation observed a 
non-significant change in the SES* achievement gap between 1995 and 2003, and 
a significant decrease thereafter. This was accompanied by a reduction in income 
inequality over the last 10 years. 


4.2.10 Singapore 


Singapore has a centralized education system and offers six years of compulsory 
education for all children. The Singaporean school system is structured by grade. 
Grades one to six are compulsory schooling. Although secondary education is not 
compulsory in Singapore, it is completed by almost every student. Grade 10 or 11 is 
the last year of secondary education (Ministry of Education, Singapore 2016). The 
majority of Singapore's students go to publicly funded schools. 

Singapore has participated in every cycle of the TIMSS assessment from 1995 to 
2015, and is consistently among the top performers in the TIMSS mathematics and 
science assessments. A very high percentage of Singaporean students in the bottom 
SES* quartile also achieved at or above the TIMSS intermediate benchmark in both 
mathematics and science. However, the achievement gap between high- and 
low-SES* students has remained large. 

The SES* achievement gap increased from 1995 to 2003, and reached a peak in 
the 2007 cycle of TIMSS in both mathematics and science (Figs. 4.23 and 4.24). For 
example, the gap in mathematics was 48 points in 1995, 85 points in 2003, and 
106 points in 2007. The achievement gap for mathematics remained a little over 
80 points during the second decade of TIMSS. Singapore clearly demonstrates a 
significantly widening achievement gap over 20 years of TIMSS. Significance test 
results also showed that the increase in the SES* achievement gap between 1995 and 
2015 (37 points) was statistically significant (Table 4.1). 

Moreover, changes in the percentage of low-SES* students achieving at or above 
the TIMSS intermediate benchmark indicated the same trend. As mentioned, 
low-SES* students in Singapore performed impressively well in this measure 
compared with students in most other education systems; this is reflected by a 
consistently high percentage of low-SES* students achieving at or above the 
TIMSS intermediate benchmarks. For example, in 1995, almost all students in the 
lowest SES* quartile achieved at or above the TIMSS intermediate benchmark for 
mathematics. However, by 2007, that number had declined substantially, even 
though it remained at a relatively high level of 74%. It increased again to almost 
85% in 2011 and 2015, which, while higher than in most other education systems, 
was still lower than it was in 1995. In summary, despite the relatively high 
achievement of the low-SES* group, the achievement gap in Singapore has 
widened over the 20 years of TIMSS, and educational equality is lower than in 1995. 

On a larger scale, GDP per person in Singapore has been increasing over the 
study period of 20 years, from around US$ 25,000 in 1995 to US$ 53,630 in 2015. In 
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Fig. 4.23 Percentage of low-SES* students achieving at or above the TIMSS intermediate 
benchmark and the average mathematics score for low- and high-SES* groups for Singapore 
(1995-2015) 
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Fig. 4.24 Percentage of low-SES* students achieving at or above the TIMSS intermediate 
benchmark and the average science score for low- and high-SES* groups for Singapore 
(1995-2015) 
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contrast, for most of the time, the percentage of total government expenditure 
devoted to education has remained at a constant level of around 20%, with the 
lowest reported expenditure being 17% in 2010. In terms of income inequality, the 
only available data from the Gini index was 42 in 1998 and 46 in 2015. In addition, a 
look at the top 10% of earners’ share of national income in Singapore confirms the 
growing trend in inequality over the study period: this figure increased from 30% of 
the total in 1995 to almost 44% in 2014. All told, while achievement remains 
relatively high, there is evidence for increasing inequality in both income and 
education outcomes in Singapore, which supports our finding of a significant 
increase in SES* achievement gaps. 


4.2.11 Slovenia 


Slovenia distributes the governance of the education system between the central 
government and schools. The Slovenian school system is structured by grades; 
children start at grade one with nine years of compulsory education ending with 
grade nine. After compulsory basic education, the next stage is two to five years of 
non-compulsory upper secondary education with different types of general 
education and vocational and technical education (Ministry of Education, Science 
and Sport 2015). 

Slovenia has participated in every TIMSS cycle since 1995 and its overall 
performance in mathematics and science has been above the international average. 
Slovenian students, regardless of their SES* background, achieved close to or above 
the TIMSS intermediate benchmark in both mathematics and science across 20 years 
of TIMSS (Figs. 4.25 and 4.26). 

We categorized Slovenia as one of the countries that observed an overall 
significant decrease in the average mathematics score difference between high- 
and low-SES* students. For both mathematics and science, there was a minor 
decrease in the achievement gap between 1995 and 2003 (Table 4.1). For instance, 
the decline in the achievement gap between high- and low-SES* students in 
mathematics between 1995 and 2003 was 10 points, a reduction from 69 points in 
1995 to 58 points in 2003. Subsequent TIMSS cycles showed no further evidence of 
a decreasing trend, but the overall difference between 1995 and 2015 (14 points) was 
statistically significant. 

As time progressed, comparatively more low-SES* students were achieving at or 
above the TIMSS intermediate benchmark. For example, in 1995, 4146 of low-SES* 
students achieved at or above the TIMSS intermediate benchmark, but this increased 
to 61% in 2015. Therefore, the 14-point reduction in the SES* achievement gap 
between 1995 and 2015 was due to an improvement in low-SES* students’ 
performance. 

The GDP per person in Slovenia has been steadily growing over the study period 
from around US$ 10,690 in 1995 to US$ 20,873 in 2015. However, the percentage 
of total government expenditure allocated to education continuously decreased, from 
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Fig. 4.25 Percentage of low-SES* students achieving at or above the TIMSS intermediate 
benchmark and the average mathematics score for low- and high-SES* groups for Slovenia 
(1995-2015) 
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Fig. 4.26 Percentage of low-SES* students achieving at or above the TIMSS intermediate 
benchmark and the average science score for low- and high-SES* groups for Slovenia (1995—2015) 
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about 1496 in 2001 to 1096 in 2013. Government expenditure on education as a 
percentage of GDP remained at more or less the same level (5% in 1995; 6% in 
2012). The Gini index for Slovenia also remained stable over time (25 in 2004; 26 in 
2014). Although there was no change in income inequality and little overall change 
in performance between 1995 and 2015, educational inequality decreased, as 
evidenced by the significant decline in SES* achievement gaps for mathematics 
over the 20 years of TIMSS. This might be related to Slovenia's proactive efforts in 
addressing achievement gaps between specific subgroups, as mentioned in OECD's 
education policy outlook for Slovenia (OECD 2016). 


4.2.12 Sweden 


Sweden has a highly decentralized education system and provides nine years of 
compulsory education for all children. The Swedish school system is structured 
around grades; grade one is the first year of compulsory schooling, while grade 12 is 
the last year of secondary education. Most of the school budgets in Sweden are 
funded by municipalities. The various Swedish institutions, municipalities, and 
central authorities implement educational policies that are in line with the 
governmental framework, while the Swedish government and parliament decide 
the outline of the national curriculum (Axelsson 2016). 

Sweden has participated in most cycles of TIMSS since 1995, with the exception 
of the 1999 cycle. There were minor reductions in the achievement gap between 
high- and low-SES* students in both mathematics and science during the first 
10 years of TIMSS, and increases in the achievement gap over the second decade 
of TIMSS (Figs. 4.27 and 4.28). As an example, the science achievement gap 
between high- and low-SES* students decreased from 85 points in 1995 to 
69 points in 2003; this, however, was not statistically significant. The science 
achievement gap subsequently increased to 73 points in 2007, 86 points in 2011, 
and 94 points in 2015. This 24-point increase between 2003 and 2015 represents a 
significantly widening SES* achievement gap. 

While the gap was closing (between 1995 and 2003), fewer low-SES* students 
were achieving at or above the TIMSS intermediate benchmark. For example, for 
science, 67% of low-SES* students achieved at or above that benchmark in 1995, 
and this figure declined to 5996 in 2003. Therefore, the decreasing achievement gap 
from 1995 to 2003 was a result of a sharp decline in both high- and low-SES* 
students’ performance, with the low-SES* group declining less severely than the 
high-SES* group. Furthermore, the percentage of low-SES* students achieving at or 
above the TIMSS intermediate benchmark increased from 4996 in 2007 and 4696 in 
2011, to 52% in 2015. However, the increase in low-SES* students’ performance 
was not as steep as it was for high-SES* students. Therefore, although more 
low-SES* students achieved at or above the intermediate benchmark, the 
achievement gap after 2007 has still been increasing. 
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Fig. 4.27 Percentage of low-SES* students achieving at or above the TIMSS intermediate 
benchmark and the average mathematics score for low- and high-SES* groups for Sweden 


(1995-2015) 
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Fig. 4.28 Percentage of low-SES* students achieving at or above the TIMSS intermediate 
benchmark and the average science score for low- and high-SES* groups for Sweden (1995-2015) 
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GDP per person in Sweden has been progressively increasing over the 20-year 
study period from US$ 29,914 in 1995 to US$ 50,812 in 2015. Meanwhile, more 
public expenditure has been focused on education. The percentage of total 
government expenditure allocated to education has been continuously growing, 
from 1096 in 1995 to almost 1596 in 2014. Likewise, the percentage of СПОР spent 
on education has increased from 696 in 1995 to almost 896 in 2013. The trend in the 
Gini index for Sweden (26 in 2004; 27 in 2014) suggests there were no major 
changes in income inequality over that time period. However, we observed a 
statistically significant increase in the SES* achievement gap for science in the 
second decade of TIMSS, which was unexpected given Sweden's increased 
investment in education. Analyzing and understanding this discrepancy seems an 
important topic for future research efforts. 


4.2.13 United States 


The United States provides compulsory education for all children, varying by state, 
from between five years old to as high as 18 years old. In the United States, there is 
no nationwide education curriculum. Local school districts and state education 
agencies are responsible for subject area curriculum frameworks (US Department 
of Education 2008). Typically, kindergarten or grade one is the first year of 
compulsory schooling while grade 12 is the last year of secondary education. 
Funding for public education in the United States is provided by federal, state, and 
local governments, and public schools are free for all children from kindergarten 
through grade 12. 

The United States has participated in every cycle of the TIMSS assessment since 
1995. 'The United States was the only country that experienced an overall significant 
decrease in the SES* achievement gap for science between high- and low-SES* 
students over the 20-year period studied (Fig. 4.29). While there was no statistical 
difference in the achievement gap between high- and low-SES* students in science 
between 1995 (105 points) and 2003 (93 points) and the decrease of 11 points in the 
SES* achievement gap between 2003 and 2015 was also not statistically significant, 
the overall decrease of 22 points from 1995 to 2015 (82 points) was significant. 
SES* achievement gaps for mathematics displayed similar patterns, although the 
difference was not significant (Fig. 4.30). 

Around 4596 of low-SES* students achieved at or above the TIMSS intermediate 
benchmark in the United States for science in both 1995 and 1999. This increased to 
more than 50% in 2003, 2007, and 2011. By 2015, almost 60% of low-SES* 
students were achieving at or above the TIMSS intermediate benchmark, 
suggesting a continuing improvement in low-SES* students’ performance in the 
United States, and providing an explanation for the decrease in the SES* 
achievement gap. To conclude, the United States is an example of a country that 
has seen an improvement in the science performance of its most disadvantaged 
students during the 20 years of TIMSS, as well as a slowly decreasing SES* 
achievement gap. Results for mathematics were similar. 
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Fig. 4.29 Percentage of low-SES* students achieving at or above the TIMSS intermediate 
benchmark and the average science score for low- and high-SES* groups for the United States 
(1995-2015) 
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Fig. 4.30 Percentage of low-SES* students achieving at or above the TIMSS intermediate 
benchmark and the average mathematics score for low- and high-SES* groups for the United States 
(1995-2015) 
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In the United States, GDP per person has been steadily increasing, from US$ 
28,782 in 1995 to US$ 56,469 in 2015. The percentage of GDP spent on education 
has remained around 5% over that period. However, the percentage of total 
government expenditure allocated to education continuously decreased, from 
about 17% in 2002 to 13% in 2013. The United States’ Gini index remained 
around 40 over the period 1995 to 2015. In spite of a reduction in government 
spending on education, our findings suggest there was a significant reduction in the 
SES* achievement gap across the years for science, and for mathematics the there 
was also a decline, albeit one that was not significant over the 20-year time period 
studied. Additional analyses are needed to further understand these contrasting 
results. 
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Chapter 5 м 
Trends in Socioeconomic Achievement Gaps 2 
in the Macroeconomic Context: Discussion 

and Future Research 


Abstract Using the 20 years of comparable data collected by the IEA's Trends in 
International Mathematics and Science Study (TIMSS), this study aimed to identify 
and describe possible relationships between trends in the inequality of educational 
opportunities related to the differing socioeconomic status (SES) of the students. 
Thirteen educational systems were assessed as having sufficiently comparable data 
for the purposes of this research and a modified version of the TIMSS home 
educational resources (HER) index was developed for the analyses. Country-level 
indicators, such as centralization of educational systems, gross domestic product per 
person, government expenditure on education, a country's Gini index, and the top 
10% of earners’ share of pre-tax national income, provided additional relevant 
information. The research revealed some tentative patterns that may be worthy of 
deeper investigation. In the second decade of TIMSS (2003-2015), larger reductions 
in the achievement gap between low- and high-SES students tended to accompany 
steeper increases in the overall TIMSS performance in the corresponding education 
systems. There were also some indications that this may be associated with the level 
of centralization of an educational system and trends in country-level spending on 
education; this needs to be confirmed by further research as only broad connections 
were made with such macro-level indicators. Future research could usefully analyze 
the measurement invariance of the modified SES index used in the study, or 
investigate the potential of reconstructing the established TIMSS HER index so it 
can be confidently applied to analyses across all TIMSS administrations. 


Keywords Achievement gaps - Centralized versus decentralized education 
systems - Educational inequality - Education spending · Income inequality - 
International large-scale assessment - Macroeconomic indicators - Trends in 
International Mathematics and Science Study (TIMSS) 
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5.1 Summary of the Findings 


Although the association between family 5Е5“! and student achievement has been 
thoroughly investigated in previous research (see Chap. 2), the extent of change in 
that association in individual education systems over time is less well known. 
Improving achievement among their disadvantaged students and narrowing the 
achievement gaps between students of low- and high-SES backgrounds is a 
common policy goal for many education systems. However, the lack of 
quantifiable measures, especially those that are easy to understand, makes it 
difficult to track and assess the effect of such efforts. 

Twenty years of TIMSS data, from 1995 to 2015, provide researchers with a 
means to empirically address important research questions regarding changes in 
educational inequality over time. We used the TIMSS data to examine whether the 
inequality of educational outcomes due to SES* has changed for education systems 
over time and to investigate the extent to which disadvantaged students improved 
their academic performance over time in each education system. 

Our first research question was: "How has the inequality of education outcomes 
due to family socioeconomic status changed for different education systems between 
1995 and 2015?" We created a modified version of the TIMSS home educational 
resources (HER) index that was consistent over the 20-year period to define low- and 
high-SES* groups of students. For each educational system and assessment cycle, 
we calculated the achievement gap between students in these low- and high-SES* 
quartile groups. When examining achievement gaps in either mathematics or science 
between 1995 and 2015, our results suggested that Hungary, Iran, Lithuania, and 
Singapore experienced a significantly widening gap between low- and high-SES* 
students, while Norway, Slovenia, and the United States observed a significantly 
narrowing gap. By contrast, some educational systems observed some significant 
changes in one of the two decades of TIMSS (1995—2003 and 2003-2015), but not 
in the other. For example, Australia experienced a significant decrease in the SES* 
achievement gap for science between 1995 and 2003, and then a significant increase 
between 2003 and 2015, resulting in an overall non-significant trend over the 
20-year period. Similarly, New Zealand experienced some decrease in the SES* 
gaps in the first decade of TIMSS, but this was followed by a significant increase in 
the second decade. There are many other examples where the more detailed study of 
trends broken down into the 1995-2003 and 2003-2015 time periods reveals 
interesting countervailing trends that warrant a closer look by researchers with a 
deep understanding of the local contexts. 

Our second research question was: "To what extent have education systems 
managed to increase the academic performance of disadvantaged students 


'The SES measure used in this study is a modified version of the TIMSS home educational 
resources (HER) index and does not represent the full SES construct, as usually defined by parental 
education, family income, and parental occupation. In this report, we therefore term our measure 
SES* to denote the conceptual difference (Please refer to Chap. 1 for more details). 


5.2 Relating the Findings to Country-Level Indicators in the Educational. . . 73 


between 1995 and 2015?" To answer this, we calculated the percentage of low-SES* 
students who performed at or above the TIMSS intermediate benchmark in each 
education system over time. It was of great importance to examine this question in 
conjunction with the first question because stagnant scores for low-SES* students and 
declines in the scores of high-SES* students are equally undesirable, but may also 
underlie a headline reduction in inequality. For example, in the Republic of Korea, the 
achievement gap in mathematics was 107 points in 2011 but declined to 84 points in 
2015; this was not due to an improvement for low-SES* students but rather a decline 
in the performance of their high-SES* students (Fig. 4.13). In contrast, the United 
States showed a decreasing achievement gap for science between 1995 and 2015, 
which corresponded to a continuous improvement in the performance of their 
low-SES* students (Fig. 4.30). Ideally, education systems should strive for equality 
by improving the performance of all students and by aiming to improve the 
achievement of low-SES* students at a faster rate to reduce gaps in achievement 
(Mullis et al. 2016). 


5.2 Relating the Findings to Country-Level Indicators 
in the Educational Systems and the Macroeconomic 
Context 


To better understand our findings in the larger context in which education systems 
operate, we obtained macroeconomic and other indicators from the TIMSS 
encyclopedias, as well as data from external resources (see Table 3.6 for all 
sources). Our goal was to explore changes in country level indicators over time 
and contrast them with changes in the SES* achievement gap (Table 5.1). A few 
tentative patterns emerged, which merit further investigation. 


5.2.1 Tentative Pattern 1: Reductions in the Achievement Gap 
Tend to Accompany Improvements in Overall TIMSS 
Performance 


In the second decade of TIMSS, we identified an inverse relationship between the 
trends in SES* achievement gaps and the TIMSS national averages for both 
mathematics and science. This finding was consistent with the previous literature 
using other cycles of TIMSS, which suggested a prominent inverse relation between 
the within-country dispersion of scores and the average level of scores by country 
(Freeman et al. 2010; Mullis et al. 2016). In other words, greater reductions in the 
achievement gap between low- and high-SES* students tended to accompany higher 
rates of increases in overall TIMSS performance over the 2003-2015 period (see 
Figs. 5.1 and 5.2). This is not a trivial finding, since, as discussed previously, a 
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Fig. 5.1 Difference in average mathematics score and SES* achievement gap, by education 
system, 2003-2015. (Note The Islamic Republic of Iran was treated as an outlier and not included 
when fitting the regression line) 
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Difference in average science performance 


Fig. 5.2 Difference in average science score and SES* achievement gap, by education system, 
2003-2015. (Note The Islamic Republic of Iran was treated as an outlier and not included when 
fitting the regression line) 


continuous growth in overall performance is possible without any reduction over 
time in the SES* achievement gap if both low and high SES* groups see the same 
rate of performance growth or shrinkage over time. If that were the case for the all 
studied education systems, the regression line would be flat. 
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5.2.2 Tentative Pattern 2: Education Systems That Observed 
Increases in Achievement Gaps Tend to be 
Decentralized 


While education systems that were able to reduce the SES* achievement gap could 
be either centralized or decentralized systems, almost all the education systems that 
Observed increases in their SES* achievement gaps were decentralized systems, with 
the exception of the Islamic Republic of Iran (Table 5.1); note that this education 
system was also an outlier in the previous analysis. 


5.2.3 Tentative Pattern 3: Education Systems That Reduced 
Investment in Education Tended to Observe 
an Increased Mathematics Achievement Gap 


When examining the changes in the percentage of GDP spent on education (this 
indicator spanning mostly 2003-2015 figures, see Table 5.1), results suggested that 
those education systems that saw a reduction of investment in education over time 
also happened to observe a significant increase in the SES* mathematics 
achievement gap in the second decade of TIMSS (Fig. 5.3). 
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Mathematics Achievement Gap Change (2003-2015) Changes in the Percent of GDP Spent on Education 


Fig.5.3 Changein the percent of GDP spent on education and in the mathematics achievement gap, 
by education system, 2003-2015. (Notes Indicator “Percent of GDP Spent on Education" was 
obtained from the World Bank Open Data and the UNESCO Institute for Statistics (see Table 3.6). 
The calculation of change in the percent of GDP spent on education between 2003 and 2015 is the 
difference between the percentages in 2003 and 2015 divided by the percentage in 2003) 
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5.3 Limitations of the Study and Recommendations 
for Future Research 


This study has several limitations that should be addressed by future research. The 
first limitation is that this study did not examine all potential factors that might 
explain the observed trends. Although we collected information on macro-level 
indicators in every corresponding education system over 20 years, we did not 
investigate empirically if those factors contributed to the changes in educational 
inequality that we observed. The broad connections made between macro-level 
changes and changes in educational inequality are descriptive. Future research 
exploring influential factors driving these changes would be important for 
understanding why some education systems were able to reduce the SES* 
achievement gaps and what can be learned by others. For example, multilevel 
modeling can be employed to test hypotheses regarding its potential association 
with macro-level factors. However, some of the factors may defy easy categorization 
and may be specific to individual educational systems or work only in the presence 
of other factors. Thus, there is also a place for a more contextual and qualitative 
understanding of the findings. Researchers with a deep understanding of local 
context would be in a better position to examine why and how these changes took 
place in their own educational system. 

Second, measurement invariance of common items across years could be a 
concern for the SES* index used in this study (a modified version of the TIMSS 
HER index). For example, the possession of a computer in the 1990s may carry a 
very different meaning and value to the possession of a computer in the 2010s with 
respect to what it says about a student's SES* background. We are uncertain which 
items see a drift in meaning, how much drift there is over time, and how such drift 
manifests itself in different countries. In spite of these concerns, we believe that for 
our study this was not a critical problem because we compared students in the 
highest and the lowest SES* quartile for each country and in each cycle 
separately. In other words, the meaning of the items or even the scale may change 
slightly, but this should not have had a strong influence on the comparison of the 
achievement gaps that were calculated based on the distribution of students’ SES* in 
a particular educational system and cycle. Nevertheless, future research should 
analyze measurement invariance of the SES* index itself or even reconstruct an 
item response theory scaled version of the HER index for years prior to 2011 so that 
analyses with that index would be possible across all TIMSS administrations. 

Third, it is important to recognize that the meaning of high- and low-SES* differs 
by societies. We decided to use educational system specific cut-offs to define SES* 
groups because the current study focused on the trend in educational inequality 
within a society. Therefore, in interpreting comparisons between societies, it should 
be recognized and understood that high-SES* students in one country can be very 
different from high SES* students (in an absolute sense) in another country. 

Finally, our analyses showed relatively distinct patterns of change in educational 
inequality in the first and second decades of TIMSS across countries (see Table 4.1). 
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Future research should especially focus on the second decade of TIMSS, namely the 
period 2003-2015, as many significant changes in SES* achievement gaps occurred 
in this decade. As the number of countries participating in TIMSS has expanded 
since its inception, this would have the added value of allowing more educational 
systems to be included in the analyses. Moreover, there would be a greater 
availability of comparable country level macro-economic indicators if 2003 is 
taken as the base year instead of 1995. 


5.4 What Have We Learned from Twenty Years of TIMSS 
Data? 


Over the 20 years of TIMSS, we found that only a few educational systems were able 
to significantly reduce the achievement gap between high- and low-SES* students, 
to improve the performance of their most disadvantaged students, and to increase 
their national average score. Most of the education systems that we studied did not 
observe such a promising three-fold trend. Among the 13 education systems studied, 
only Slovenia observed such a trend in mathematics and only the United States for 
science. This further reflects the difficulty of fostering positive change in academic 
performance, or maintaining high performance, for all students over time, while also 
counteracting a general rise in inequality through policies in the education system 
that would enable a closing of the SES* achievement gap and effectively address the 
needs of disadvantaged students. 

By contrast, some educational systems observed some significant changes in one 
of the two decades of TIMSS (1995—2003 and 2003-2015), but not the other. For 
example, Australia experienced a significant decrease in the SES* achievement gap 
for science between 1995 and 2003, followed by a significant increase between 2003 
and 2015. This resulted in an overall non-significant trend over the 20-year period. 
There are many other examples where a more detailed study of trends broken down 
into the 1995-2003 and 2003-2015 time periods would be of interest. Researchers 
with a deep understanding of local contexts should take a closer look at such 
countervailing trends. 

For the second decade of TIMSS (2003-2015), three tentative patterns emerged 
when contrasting changes in country level indicators over time and changes in the 
SES* achievement gaps. First, there was an inverse relationship between the changes 
in SES* achievement gaps and the changes in TIMSS national averages for both 
mathematics and science. Second, almost all the education systems with an increase 
in their SES* achievement gaps were categorized as “decentralized” education 
systems in the study. Third, the education systems with a reduction of investment 
in education happened to observe a significant increase in the SES* mathematics 
achievement gap. Although these patterns are preliminary, we encourage further 
investigation into the country-level changes with additional countries being included 
in the analyses. 
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Appendix 
Sensitivity Check 


A.1 Index Sensitivity Check 


To check the sensitivity of our results regarding the way in which the SES* index 
was constructed (simple-sum, unweighted, which allows for a straightforward inter- 
pretation of the index points), we constructed a second SES* measure based on the 
first principal component from a principal component analysis (PCA). The objective 
of РСА is to explain the variance-covariance structure of a set of variables through 
linear combinations of the variables that contain most of the variance. Although 
there are as many principal components as there are variables in the analysis, the first 
principal component always contains the most variance. In this study, the first 
principal component contained almost half of the variance (4596) and it was the 
only component with an eigenvalue (1.801) larger than 1. By using the predict 
command in the pca module in Stata 14 (StataCorp 2015), we computed the first 
principal component score for all respondents using the eigenvectors (Table A.1) 
and used it as a second measure of SES*. In short, this alternate measure of SES* is a 
weighted sum of the standardized variables: 


SES* = 0.572 х Zparent educ + 0.432 X Zeomputer + 0.520 х Zpooks + 0.465 X Zaesk 


We generated education system and year-specific cut-offs for the principal 
component score and compared the trend results based on the two different mea- 
sures. The results suggested that the way the SES* measure was constructed did not 
change the general trend that we observed for the countries that showed significantly 
widening mathematic achievement gaps between high- and low-SES students 
(Fig. A.1). This was expected, given that, according to Wainer (1976), choosing a 
certain weighting scheme of the components in a sum score over another should in 
general have limited impact if the components are correlated. However, results could 
still be sensitive to the way high- and low-SES* groups were constructed. We thus 
performed a cut-off sensitivity check. 
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Table A.1 Principal component analysis of the SES* index 


Eigenvalues Coefficient | Standard error | p>t | [95% confidence intervals] 
Component 1 1.801 0.005 0.000 | 1.791 1.810 
Component 2 0.844 0.002 0.000 | 0.840 0.849 
Component 3 0.778 0.002 0.000 | 0.774 0.782 
Component 4 0.577 0.002 0.000 | 0.574 0.580 
Principal component score 
Parental education 0.572 0.001 0.000 | 0.569 0.574 
Computer 0.432 0.002 0.000 | 0.428 0.437 
Number of books at home | 0.520 0.002 0.000 | 0.517 0.523 
Study desk 0.465 0.002 0.000 | 0.461 0.468 
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Fig. A.1 Increasing mathematics score difference between high- and low-SES* students using 
SES* measure constructed by PCA, by education system and year 


A.2 Cut-off Sensitivity Check 


As an additional sensitivity check, we compared the impact of choosing a different 
definition of low- and high-SES* groups by using quintiles instead of quartiles as 
cut-offs. In general, we observed minor differences between these two approaches, 
and none that would change the general trend pattern. For mathematics, we repeated 
the analysis of the percentage of students in the low-SES* groups performing at or 
above the TIMSS intermediate benchmark (based on quartiles) with the SES groups 
now defined by quintile cut-offs. On average, among the 13 education systems 
across all cycles of TIMSS, using quintile cut-offs resulted in about a 2% reduction 
in the percentage of students achieving at or above the TIMSS intermediate 
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Differences in percentages 


Education system and year 


Note The calculation subtracts quintile cut-off from the quartile cut-off for the percentages of low-SES* students 
performing at or above the TIMSS mathematics intermediate benchmark. 


Fig. A.2 Differences in percentage of low-SES* students achieving at or above the TIMSS 
mathematics intermediate benchmark: quartile versus quintile cut-offs, by education system 
and year 


benchmark. No change went beyond a four-percentage point difference (see 
Fig. A.2). Therefore, we concluded that using quintile cut-offs would not have 
changed the patterns of our findings in any fundamental way. 
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